Sunday, April 12, 2015

Cooking at Home or Eating Out? - The Pros and Cons of Homegrown VIP

By now, we're all pretty much convinced that reuse is essential in the semiconductor industry. Gone are the days when we built everything from scratch an re-invented the wheel on each project. Today, to build a new SoC we stitch together multiple blocks that we've already used on previous projects, maybe tweaking them a little, while only designing those new things that will help our product differentiate itself from the competition.

The same principle also holds for the testbenches we develop for those blocks and SoCs. It makes sense, right? If we have a block with an AHB interface, do we really need to write a verification component that can drive the AHB protocol for each new iteration of that block? What if we have multiple blocks that communicate through AHB? Why couldn't the same verification component be used within all of those testbenches? The industry recognized this and some time ago new languages appeared that borrowed from general purpose programming languages to make it easier to develop reusable verification intellectual property (VIP).

These days, we have two choices when considering VIP. We can either buy commercial products from established vendors or we can take charge and implement these ourselves. There are many ideas scattered throughout the net on the make or buy dilemma as it applies to VIP. Most of them were written by VIP vendors and really, what are they going to say? I don't claim any authority on the topic with this post, but I want to share my view as both a homegrown VIP developer and subsequent user.

A little while back our team decided to go the way of the warrior and develop our own in-house UVCs for the AMBA protocols we use. This was by no means anything new within the company. We already had a considerable portfolio of eVCs and other UVCs (of varying quality) for various protocols our chips employ, both proprietary and standard. I was lucky enough to have just finished a project around that time and to get a chance to be involved in the development. Now, more than a year later, I've started to use some of them on my current project. This is a good time to reflect on the pros and cons of homegrown VIP.

The most obvious advantage to building versus buying is the financial one. The cost model I've seen up to now for VIP involves subscription fees for licenses. This is money that can be saved. We do have to pay upfront by having to spend engineering time developing the VIP, though. This can lead to somewhat of a ski rental problem because we might not really be able to say how high this initial cost will be, but for simple protocols I'd argue this isn't such a big deal. For example, I'd say that a UVC for a simple protocol like APB could be developed in about one man-week and one for something more complicated like AHB maybe in something of the order of one man-month. Protocols of the complexity order of AXI and beyond are a whole different matter...

While money is an important factor, that wasn't the reason we went down the route of writing our own UVCs. The main one was stability. I think a lot of us have seen the case where VIP from vendor X doesn't work with the simulator from vendor Y. We know that the SystemVerilog standard isn't specific enough and open to interpretation in some places and different simulators can treat a construct in different ways. At the same time, the degree to which different vendors have implemented the standard also varies, with some constructs being supported in one tool, but not in another. Having the source code under our control means that we can tackle such issues, so switching simulator vendors won't impact us so dramatically anymore. But having write access to the source code can be both a blessing and a curse...

We have to admit that more often than not, when starting something new we get very excited and want to jump directly into our text editors and start writing some code. This "code first, think later" mentality can only hurt us because instead of focusing on quality from the beginning we might just say "we can weed out the bugs later". Bugs in a UVC can be disastrous as they can lead to design bugs escaping the net and making it onto silicon. This isn't to say that commercial VIP is 100% bug free (as I've seen my fair share), but vendors are more likely to have good quality control procedures set up, meaning that bugs become rare. Unit and integration testing are the tools we can use to mitigate such problems. If you haven't already I invite you to have a look at SVUnit. I've also written about it in this post. If there's anything that really, really, really needs be properly tested, it's our UVCs!

Another blessing when we have access to the source code of our UVCs (and not jus read-only) is extendability. It's much easier to accommodate deviations from the standard protocol when we can build these directly into the UVC and not have to bolt them onto a standard UVC. How we do this, however, should depend on whether or not we need to implement the standard protocol too.

The curse begins to manifest itself when we would like to have our UVC do some cool thing that we only need in our current project, but that no one else needs. Since we have the source code under our command, we might think that it won't do any harm to add new capabilities to the UVC. They may be useful to others someday too! It's also easier that using OOP concepts like inheritance and/or composition, so the temptation is just too big. What we get with such feature creep, though, is a tangled mess of configuration switches, extra interface signals that don't exist in the specification and spaghetti code for our UVC components. What we need is a clear vision of what is of general interest and what is not. We can use our power over the code for good, by adding hooks to it to facilitate extendability, but not to implement everything on a whim.

Homegrown UVCs will also tend to have ad-hoc maintenance practices. This is also coupled with lax release processes, with no guidelines for deprecating old features and adding new ones. My biggest pet peeve is source code management, where everything will get dumped into the main branch, instead of making use of branches and labels (or whatever the terminology of your SCM is) and other tools that are provided to us. The main question here is "do we release tar files with specific versions or do we point to the repository where the UVC development is done?". This, in my opinion, is the root of all evil. In the latter case, users will get itchy fingers and want to change code to fit their needs. As we saw above, even when done with the best intentions, this isn't a good idea. I tend to avoid this even for fixes, even if I do have access to the source code and the development repository. If I need to tweak the behavior of the UVC I prefer to create extended classes in my verification environment rather than change something in the UVC itself. These can always be incorporated later or be released as a standalone package for others to use.

Last but not least, since documentation is always a second class citizen, it will just fall behind (I plead guilty here!). Even if we spend time to do the documentation properly and to keep it up-to-date, people still won't read it if they have someone to ask in person. To prove this, how many times have you asked your simulator's application engineer about something instead of reading up on it in the manual?

Lately, open source UVCs have started appearing. The guys over at AMIQ Consulting have already released three VIP packages: Ethernet, APB and DCR. I haven't looked at them, so I won't comment, but the company does have an EDA arm hence I'm guessing they understand development concepts much better than I do (and they could probably even add to the points above). I'm hoping to see many more such such releases in the future, from them and from others. I mentioned open source UVCs here since they have the potential of saving us the initial development cost associated with building our own, while still providing all of the advantages. They also, however, carry the same temptations. Remember, having write access to the source code is power and, as we know already, power corrupts!

In conclusion, having homegrown VIP can be a really good thing, as long as we invest the time to do it right. The list ended up having more cons than pros, but these can be mitigated. We just need to plan changes appropriately and not just commit every single thing into the UVC repository without thinking and especially not without testing. It might not seem worth it to spend the extra effort, but if we don't we'll end up using a lot of our time fire-fighting and we may have just been better off buying the VIP. We need to set responsibilities for the UVC's source code and stick with them. We need to know that we have two hats to wear, one as a customer and one as a developer, and we need to know when to wear each.

6 comments:

  1. Hi Tudor,

    Nice post. While consulting with various companies I have come to a conclusion that the decision is also dependent on the DNA - the culture of the company. On other aspects I did write something on this subject too -

    http://whatisverification.blogspot.in/2013/08/verification-ip-build-or-buy.html
    http://whatisverification.blogspot.in/2012/03/choosing-right-vip.html
    http://whatisverification.blogspot.in/2013/10/essential-ingredients-for-developing.html

    Best Wishes
    Gaurav Jalan

    ReplyDelete
    Replies
    1. For sure a lot of teams/companies have a dislike for all things NIH (not invented here).

      Delete
  2. Why limit yourself to one or the other? Why not the best of both world using both home grown and 3rd party VIP. Nope, I am not talking about pick the right VIP based on ROI on different interfaces. In fact we have both 3rd party and home grown VIP (same protocol) in the same testbench hooking up to the same interface. Depending on the test requirement, we can choose the most cost effective VIP for the job.

    For example, if I need to test AXI compliance of the block, the testbench will instantiate the 3rd party VIP to stress test the bus. For most of the testcases that uses AXI as a dumb register bus, we will use the light weight home grown AXI VIP to save license cost.

    Implementing our light weigh AXI VIP is easy and cheap as long as we limit the supported features and let the 3rd VIP do the heavy lifting. Moreover we can use the 3rd party VIP to QA our homegrown VIP.

    There is also a very important intangible benefit of building light weight homegrown VIP. The task is a provides very good hands on training in verification methodology and in-depth ramp up of the protocol domain knowledge to the less experienced verification engineers.

    ReplyDelete
    Replies
    1. That's a really interesting take on it, have your cake and eat it too. If you're starting your homegrown lightweight VIP from a solid base and adding to them, they could even have the potential to replace the 3rd party solutions completely in the long term.

      Delete
    2. @hevangel: Beside your example I would also consider: low complexity protocols(e.g. APB) or VIPs that support a protocol's upper layers that can stack on top of 3rd party VIPs (e.g. for example a light TCP/IP VIP stacked above a 3rd party Ethernet VIP). The second one is especially efficient when there is no need for full coverage of the protocol stack given that a silicon-proven rtl IP is used.
      In case you are not interested in data collection you can develope an SVA package instead of a VIP.
      As a side note: might be easy to develop a VIP, but than you have to qualify the VIP. This kind of effort I could see many times being ignored or minimized by the VIP developers and project managers alike.

      @tudor:
      On protocol exceptions:
      I think it's a good option to use branches or overwriting classes in the local environment instead of forcing exceptions into the VIP.

      On VIP ownership:
      There should always be ONE engineer assigned to be the owner of one or more VIPs, which means it is the one to decide what should be updated, added or removed. The VIP owner handles the release process and mediates change requests.

      On proprietary VIPs:
      If the protocols are part of the company's IP and sensitive to outsourcing, most probably the company will go for a homebrew solution, which does not exclude consultants doing it.
      Also a home brew VIP is not free of risks: the more complex the protocol the higher the risks of not being standard/protocol aligned, schedule slips, simulator portability issues.

      On replacing a 3rd party IP by an internal VIP:
      Companies have to balance between internal sourcing and 3rd party IP procurement or VIP development outsourcing. VIP development,qualification&maintenance is not part of the core business of an ASIC/SoC company and in the long run can be dead weight.

      Nowadays many of the ASIC/SoC companies are integrating IPs and not designing those from scratch. These companies MUST focus their QA efforts on the integration verification and not IP verification. Given that, the company will not care too much of the small details of the protocols supported by the IPs and then they can use a lightweight VIP or rent a VIP for a limited amount of time.

      I am confident that for complex protocols (e.g. USB, DDR, AHB, Ethernet) it's more efficient (e.g. considering time-to-market, risk reduction, inventory) to rent VIPs, rather than developing those from scratch. It's a heavy project to develop, qualify and maintain a DDR4 VIP. Leaving aside development, which is more or less similar to software development, you have to seriously consider qualification and maintenance effort. "Silicon-proven" label is a quality standard, either we like it or not, and that is why qualification might require silicon proven references, which means you will have to "rent" a silicon proven IP. One should consider that dependencies updates (e.g simulator, UVM, SystemVerilog Standard) trigger qualification sessions for every update increasing the maintenance effort.

      I think the "internal vs 3rd-party" decision should seriously consider the time-to-market, qualification&maintenance effort, associated risks, next technology update, dependency associated risks, not just how much it takes to develop it.

      On AMIQ VIPs:
      Those were released for the open public to use. there will probably be updates, but without any commitment.

      Delete
    3. @Stefan I agree that someone should be put in charge of a piece of VIP and analyze any potential changes to it prior to implementing them. What you have to be careful of is the case where the owner is also a user of the VIP and adds stuff to it that is only relevant for their project instead of implementing such functionality inside their verification environment. That's why we moved to OOP, to be able to separate the generic from the specific.

      W.r.t. buy vs. build, this post wasn't written in the spirit of convincing people to build their own VIPs. I just wanted to warn anyone that already made this decision in favor of 'build' what to look out for, based on my experience.

      Delete