SPEAKER_01: As context windows get larger and larger,
SPEAKER_01: one question that keeps coming up
SPEAKER_01: is whether retrieval augmented generation
SPEAKER_01: or RAG is becoming obsolete.
SPEAKER_01: If models can ingest millions of tokens of context
SPEAKER_01: and reason over enormous collections of documents,
SPEAKER_01: why bother with retrieval at all?
SPEAKER_01: The answer, it turns out,
SPEAKER_01: depends a lot on the application.
SPEAKER_01: I recently sat down with Alex Boca,
SPEAKER_01: head of engineering at Sphere,
SPEAKER_01: which builds AI systems for sales tax, automation,
SPEAKER_01: and compliance.
SPEAKER_01: Exactly the kind of domain we're getting
SPEAKER_01: the right answer isn't enough.
SPEAKER_01: You also need to know where it came from.
SPEAKER_01: I asked him this simple question.
SPEAKER_01: What's your take on the whole RAG is dead,
SPEAKER_01: argument that some folks make?
SPEAKER_01: I think for some use cases, it's certainly true.
SPEAKER_01: I think for us, or at least for this particular problem,
SPEAKER_01: because we are so sensitive to accuracy
SPEAKER_01: and we're so sensitive to the exact right citation.
SPEAKER_01: As of today, I don't think agents
SPEAKER_01: are just searching over the file system
SPEAKER_01: and gripping over it is at a point where
SPEAKER_01: we could switch over and not lose accuracy.
SPEAKER_01: I'm Sam Charrington, and this is the Tumel AI podcast.
SPEAKER_01: For over a decade, I've been exploring the ideas
SPEAKER_01: and innovations, shaping the future of AI
SPEAKER_01: through conversations like this one
SPEAKER_01: that help you understand what's real,
SPEAKER_01: what's next, and what matters.
SPEAKER_01: Let's jump in.
SPEAKER_01: A little bit about Sphere briefly.
SPEAKER_01: So this makes a little more sense.
SPEAKER_01: Sphere is a revenue-based compliance company.
SPEAKER_01: So we help companies with all of their revenue-based compliance
SPEAKER_01: needs.
SPEAKER_01: The main one of those is sales tax in the US.
SPEAKER_01: And internationally, that's called VATGST.
SPEAKER_01: And the way there's other companies in this space, of course,
SPEAKER_01: there's some big companies that have been around for quite a while.
SPEAKER_01: Tax is not a new problem.
SPEAKER_01: It's not, unfortunately.
SPEAKER_01: For companies and for consumers, I suppose.
SPEAKER_01: So this isn't a new problem.
SPEAKER_01: They face a particular problem, which
SPEAKER_01: is in order to support every jurisdiction in the US,
SPEAKER_01: because of course, every US state has different rules.
SPEAKER_01: In some US states, even the cities have different rules.
SPEAKER_01: And then internationally, of course, every country and potentially province
SPEAKER_01: has their own rules as well.
SPEAKER_01: And so the companies need, the incumbents
SPEAKER_01: need a way to understand how our products
SPEAKER_01: taxed in each of these different jurisdictions.
SPEAKER_01: And the way that traditionally they've
SPEAKER_01: done these sorts of things is they've hired these massive teams
SPEAKER_01: of essentially tax lawyers, their tax experts.
SPEAKER_01: They'll call them tax content teams.
SPEAKER_01: What these tax lawyers are doing, essentially,
SPEAKER_01: is looking through the legislation in Alabama, for example,
SPEAKER_01: and understanding how does Alabama tax SaaS?
SPEAKER_01: And even more specifically than that,
SPEAKER_01: how does Alabama tax SaaS that maybe has an API connection
SPEAKER_01: and has servers that are hosted within the state itself?
SPEAKER_01: So it gets very granular there.
SPEAKER_01: And this takes a huge amount of human time to do.
SPEAKER_01: And there's also the moving target of it,
SPEAKER_01: which is, of course, like legislation updates
SPEAKER_01: that can happen at any moment.
SPEAKER_01: And so you have to constantly be updating
SPEAKER_01: and looking through the legislation again
SPEAKER_01: to see if anything has changed
SPEAKER_01: and updating your tax engine, essentially,
SPEAKER_01: to make sure that you're applying the correct treatment
SPEAKER_01: in all of the jurisdictions.
SPEAKER_01: And so that's been a huge inhibitor to growth
SPEAKER_01: for the incumbents.
SPEAKER_01: And the reason why most of the big incumbents
SPEAKER_01: have stayed in the US because they've kind of tackled
SPEAKER_01: this problem in the US and to extend it internationally,
SPEAKER_01: it's just like too much of a herculean task for them
SPEAKER_01: to, it's too much manpower, too big of teams to handle it.
SPEAKER_01: And Sphere has taken a very different approach.
SPEAKER_01: I think the time that Sphere was started as a company
SPEAKER_01: was obviously advantageous.
SPEAKER_01: We were started during the AI era.
SPEAKER_01: And so this is a very classic document-based problem.
SPEAKER_01: It just, from a high level, you have legislation
SPEAKER_01: and court rulings and bulletins
SPEAKER_01: from departments of revenue.
SPEAKER_01: These are all just documents.
SPEAKER_01: And these inform the answers of how products are taxed
SPEAKER_01: or found in these documents.
SPEAKER_01: And it's just a matter of finding the relevant passages
SPEAKER_01: and understanding the relevant passages
SPEAKER_01: and then assigning a tax ability.
SPEAKER_01: How is this product taxed?
SPEAKER_01: And so in this new era that we're in,
SPEAKER_01: we looked at that problem.
SPEAKER_01: And it was a problem that we thought
SPEAKER_01: was screaming to be solved by AI.
SPEAKER_01: So what we eventually built is what we call T-RAM,
SPEAKER_01: which is the tax review and assessment model,
SPEAKER_01: which is like a system of a few different things
SPEAKER_01: that I'm sure we'll get into.
SPEAKER_01: But essentially, its job is to supercharge our tax experts.
SPEAKER_01: So what we found is that T-RAM allows our internal tax
SPEAKER_01: experts to move almost two orders of magnitude
SPEAKER_01: faster through this process with less errors
SPEAKER_01: than the traditional, just fully human focused approach.
SPEAKER_01: And did you work in tax part to joining Sphere?
SPEAKER_01: I did not.
SPEAKER_01: So I've learned a lot about tax.
SPEAKER_01: And I think I was always like, I would read US Supreme Court
SPEAKER_01: rulings just for fun, now to the interest of what's
SPEAKER_01: going on and how does the legal world work.
SPEAKER_01: But no, I didn't come from a traditional PWCE tax
SPEAKER_01: background or anything like that.
SPEAKER_01: What's your background?
SPEAKER_01: Yeah, just a pure engineering background, mostly in startups.
SPEAKER_01: So I started my career before moving into startups.
SPEAKER_01: I started in the semiconductor industry
SPEAKER_01: and then working on GPUs, writing CUDA kernels.
SPEAKER_01: And then eventually, left to start a startup with a friend,
SPEAKER_01: which was like a web data collection startup.
SPEAKER_01: And we worked with investment banks and private equity firms
SPEAKER_01: and things like this to help them collect really massive amounts
SPEAKER_01: of web data at scale to power their internal analyses
SPEAKER_01: and signals.
SPEAKER_01: And eventually, that company got acquired by Bain and Company.
SPEAKER_01: And so I was at Bain for a bit and then
SPEAKER_01: started another startup.
SPEAKER_01: And eventually, Nick reached out, who's
SPEAKER_01: the founder of Sphere, Nick Rudder.
SPEAKER_01: Nick reached out about Sphere.
SPEAKER_01: And at the time, it was just Nick and a little one person
SPEAKER_01: office here in San Francisco.
SPEAKER_01: And looking back, maybe a little inadvisable,
SPEAKER_01: but he convinced me it was a good idea
SPEAKER_01: to leave the other place.
SPEAKER_01: How was that?
SPEAKER_01: And come and join him at Sphere and kind of chase this dream.
SPEAKER_01: That was, he had like this nascent idea
SPEAKER_01: of what would become T-RAM.
SPEAKER_01: And yeah, things have gone quite well.
SPEAKER_01: We did our series A last year from injuries and horowitz
SPEAKER_01: and continue to grow at kind of a remarkable level.
SPEAKER_01: Talk a little bit about the data landscape
SPEAKER_01: that you have to do with.
SPEAKER_01: I'm imagining a significant amount of complexity
SPEAKER_01: due to the global nature of it.
SPEAKER_01: And whenever I've talked to folks that
SPEAKER_01: are doing a collection of legal data,
SPEAKER_01: like always surprises me how much of that stuff is in
SPEAKER_01: like non-friendly formats, like photo image-based PDFs
SPEAKER_01: or I've talked to folks.
SPEAKER_01: It's been a little while, but they had
SPEAKER_01: to go send people to go scan stuff.
SPEAKER_01: How crazy is it still?
SPEAKER_01: Yeah, it's not great.
SPEAKER_01: It's not great.
SPEAKER_01: And a lot of what we do is working
SPEAKER_01: with government systems, obviously, which are, you know,
SPEAKER_01: they can be archaic.
SPEAKER_01: There's some that are better than others,
SPEAKER_01: but yeah, there's a lot that are quite old.
SPEAKER_01: So sometimes things are well structured.
SPEAKER_01: There are HTML pages that we can go and collect
SPEAKER_01: the legislation, and that's great.
SPEAKER_01: Sometimes there are PDFs, but they're well-structured PDFs.
SPEAKER_01: And so we can parse those easily.
SPEAKER_01: And so it's always great when the sources are like that.
SPEAKER_01: And then yeah, there's this long tail of PDFs,
SPEAKER_01: but they're just images, right?
SPEAKER_01: So you need those, CR them, and use other techniques
SPEAKER_01: like that, or even spreadsheets, or text documents,
SPEAKER_01: or word documents.
SPEAKER_01: These things are much more common than I think I had hoped
SPEAKER_01: when I started working on the issue.
SPEAKER_01: And so, yeah, the beginning of our pipeline,
SPEAKER_01: like this data collection process,
SPEAKER_01: accounts for each of these different file formats.
SPEAKER_01: But yeah, not the fun is problem to solve
SPEAKER_01: and trying to, especially spreadsheets and things like this,
SPEAKER_01: quite difficult to try and pull good information
SPEAKER_01: and retain contacts when things are stored in a spreadsheet.
SPEAKER_01: Let's take a step back and dig into how your users use T-RAM.
SPEAKER_01: You mentioned that you two orders of magnitude,
SPEAKER_01: you know, two orders of magnitude,
SPEAKER_01: near improvement in their process.
SPEAKER_01: What is that process?
SPEAKER_01: And how are they using the system?
SPEAKER_01: Yeah. So our end users,
SPEAKER_01: so if you look at our website,
SPEAKER_01: folks like Lovable or Replet,
SPEAKER_01: who use Sphere, who are customers of Sphere,
SPEAKER_01: they're not direct users of T-RAM.
SPEAKER_01: T-RAM is an internal tool that allows Sphere,
SPEAKER_01: that has a loud Sphere to expand globally,
SPEAKER_01: even though we're a small company.
SPEAKER_01: We're just a startup.
SPEAKER_01: And then also have higher accuracy.
SPEAKER_01: And so the way that T-RAM is used is by our tax experts.
SPEAKER_01: So there's a web app that the tax experts use,
SPEAKER_01: where they go in and essentially just review the work
SPEAKER_01: that T-RAM is done.
SPEAKER_01: So there'll be a queue of work that needs review.
SPEAKER_01: So for example, maybe they need to review, you know,
SPEAKER_01: T-RAM is done what we call determinations,
SPEAKER_01: which is just, you know,
SPEAKER_01: this determining of whether a product is taxable or not
SPEAKER_01: and some other features around there,
SPEAKER_01: you know, particular jurisdiction.
SPEAKER_01: So the tax experts will go in and say,
SPEAKER_01: oh, I see that California digital goods
SPEAKER_01: needs to be reviewed.
SPEAKER_01: And so there's a list of different types of digital goods
SPEAKER_01: and the model's output on whether it's taxable
SPEAKER_01: or not some reasoning the model gives
SPEAKER_01: on why it came to that conclusion.
SPEAKER_01: And then also importantly,
SPEAKER_01: the citations that the model used
SPEAKER_01: in order to inform its decision there as well.
SPEAKER_01: And so the tax experts are able to look through
SPEAKER_01: really quickly and you know,
SPEAKER_01: sometimes they do have to make adjustments.
SPEAKER_01: The model isn't 100% accurate,
SPEAKER_01: but essentially they review the model's outputs,
SPEAKER_01: they can make feedback, things like that.
SPEAKER_01: And eventually they click submit
SPEAKER_01: and the submission of that then puts those
SPEAKER_01: into our deterministic tax engine.
SPEAKER_01: So we have a tax engine.
SPEAKER_01: That for example, we integrate with Stripe
SPEAKER_01: or a first party integration with Stripe.
SPEAKER_01: So our customers, you know,
SPEAKER_01: when you're checking out and buying something
SPEAKER_01: on the checkout page, we'll be calculating tax.
SPEAKER_01: And that part of it is deterministic.
SPEAKER_01: There's no AI there.
SPEAKER_01: It's the AI has been done upstream from there.
SPEAKER_01: So when you're, these tax experts are,
SPEAKER_01: you know, working on a piece of work.
SPEAKER_01: Like what's the impetus for that?
SPEAKER_01: Is it, you know, there's, you know,
SPEAKER_01: they are, you know, solving a problem for a customer
SPEAKER_01: and that like, you know, drives work
SPEAKER_01: or is the work all driven from, you know,
SPEAKER_01: some new, you know, ingested piece of data
SPEAKER_01: from a jurisdiction that the system says,
SPEAKER_01: oh, this might have a change in, you know,
SPEAKER_01: the deterministic engine.
SPEAKER_01: So it would be twofold.
SPEAKER_01: So one thing that would be like the impetus for them
SPEAKER_01: to go in would be we're expanding products
SPEAKER_01: that we want to cover.
SPEAKER_01: So every tax engine has done this.
SPEAKER_01: You can't support every product type from day one, right?
SPEAKER_01: You kind of have to break down
SPEAKER_01: and, you know, we're going to start with clothing
SPEAKER_01: or we're going to start with SaaS.
SPEAKER_01: Sphere specifically, we started with like electronic services.
SPEAKER_01: So anything that's not tangible essentially.
SPEAKER_01: So they would go through, we have this backlog essentially.
SPEAKER_01: We're like, hey, we want to add support
SPEAKER_01: for all of intangible goods, which we've already done.
SPEAKER_01: Once we've done that, now we're like, okay,
SPEAKER_01: we want to add support for tangible goods.
SPEAKER_01: So we'll move through clothing and servers
SPEAKER_01: and things like this.
SPEAKER_01: So there's like this backlog
SPEAKER_01: and that's kind of driven by customer demand, right?
SPEAKER_01: If we want to sell, if we want to sell
SPEAKER_01: to a clothing provider, of course, we need to support clothing.
SPEAKER_01: Then the other thing would be updates in the law.
SPEAKER_01: So when legislation is changed
SPEAKER_01: or new bulletins are posted or new case law becomes posted,
SPEAKER_01: then we'll scrape that data.
SPEAKER_01: It goes through our ingestion process
SPEAKER_01: eventually ends up through the system
SPEAKER_01: with if there's some action that needs to be taken,
SPEAKER_01: we'll make that recommendation to our tax experts.
SPEAKER_01: So an example might be last year, Maryland,
SPEAKER_01: or maybe a better example, Manitoba in Canada,
SPEAKER_01: they changed, they began to tax SaaS
SPEAKER_01: at the beginning of 2026.
SPEAKER_01: That was flagged very early in our system.
SPEAKER_01: And so that was reviewed
SPEAKER_01: and essentially pushed into the deterministic tax engine
SPEAKER_01: with a start date of January 1st
SPEAKER_01: so that we were prepared well in advance.
SPEAKER_01: And that's one of the nice things about having
SPEAKER_01: something like T-Rab, this automated system
SPEAKER_01: is a lot of the time traditionally
SPEAKER_01: these things are done retroactively
SPEAKER_01: because you miss the update
SPEAKER_01: and then you scramble to try and add it,
SPEAKER_01: but it's already too late.
SPEAKER_01: It's past January 1st or whatever.
SPEAKER_01: So yeah, those would be the two different ways.
SPEAKER_01: God, I got, I remember getting a flurry of emails
SPEAKER_01: maybe I don't know, nine months a year ago
SPEAKER_01: from like every SaaS vendor I use
SPEAKER_01: talking about some big change
SPEAKER_01: and the way taxes were going to be calculated.
SPEAKER_01: Yeah, I mean, it's especially on the SaaS front,
SPEAKER_01: over the last four or five, six years,
SPEAKER_01: U.S. states and international jurisdictions
SPEAKER_01: are just getting more, you know,
SPEAKER_01: they want a piece of the cake, right?
SPEAKER_01: They want to share the pie.
SPEAKER_01: And so they're changing rules to tax assets
SPEAKER_01: and they're also changing lots of other rules.
SPEAKER_01: There are certain international countries
SPEAKER_01: that require like real-time reporting
SPEAKER_01: to the department of revenue or the tax authorities.
SPEAKER_01: So like as you're transacting, they want a copy of it.
SPEAKER_01: And I think, you know, it's kind of like tax rates
SPEAKER_01: like tax rates only go up, right?
SPEAKER_01: They vary infrequently.
SPEAKER_01: Well, we'll like California reduce their sales tax, right?
SPEAKER_01: And I think how involved the tax authorities
SPEAKER_01: want to be in transactions
SPEAKER_01: and how much information they want.
SPEAKER_01: That also only increases.
SPEAKER_01: It's not going to decrease.
SPEAKER_01: And so that's something sphere I think
SPEAKER_01: is kind of at the forefront as well
SPEAKER_01: or these other features outside of just sales tax
SPEAKER_01: that are only becoming more common.
SPEAKER_01: And a big part of that is SaaS, kind of like you mentioned.
SPEAKER_01: So these human experts, in some ways,
SPEAKER_01: what it sounds like they're doing is data labeling.
SPEAKER_01: Did you think of it like that?
SPEAKER_01: We think of it more as like a legal review.
SPEAKER_01: So in some ways, I think what we do
SPEAKER_01: is somewhat similar to someone like Harvey.
SPEAKER_01: So Harvey AI, that's like a legal AI
SPEAKER_01: and you'll let Harvey draft like a first version
SPEAKER_01: of a brief maybe.
SPEAKER_01: But then a lawyer or someone at the firm
SPEAKER_01: will go in and they'll review that brief
SPEAKER_01: and they'll check it for correctness and things like that.
SPEAKER_01: They'll do a legal review.
SPEAKER_01: And I think that's how we view what our tax experts are doing.
SPEAKER_01: It's not necessarily data labeling.
SPEAKER_01: It's a review of correctness from a legal perspective
SPEAKER_01: because these are legal claims essentially
SPEAKER_01: that we're making, we are claiming that SaaS
SPEAKER_01: is not taxable in Alabama or whatever.
SPEAKER_01: So yeah, I think it's more akin to a legal review.
SPEAKER_01: And so talk a little bit about the kind of the pipeline
SPEAKER_01: in more detail you ingest this information
SPEAKER_01: from lots of different jurisdictions presumably
SPEAKER_01: normalize it in some kind of way
SPEAKER_01: or at least try to extract the information
SPEAKER_01: out of those image PDFs.
SPEAKER_01: What happens next?
SPEAKER_01: Yeah, once we have kind of the text from the document
SPEAKER_01: and ideally that's well structured
SPEAKER_01: and I gauge email and some PDFs will give that to you.
SPEAKER_01: So we try and preserve as much structure as possible.
SPEAKER_01: The next step would be for non-English legislation,
SPEAKER_01: we'll do an English translation.
SPEAKER_01: So that's been another big unlock
SPEAKER_01: is that we don't need tax experts
SPEAKER_01: that speak every language.
SPEAKER_01: LLM's notoriously great translators
SPEAKER_01: and so they're happy to translate these documents.
SPEAKER_01: So we'll create an English translation
SPEAKER_01: and that's kind of the starting point.
SPEAKER_01: From there, many of these documents are very long.
SPEAKER_01: So we can't just take the whole document
SPEAKER_01: and create an embedding for it
SPEAKER_01: and store that in a vector database
SPEAKER_01: or even necessarily with a TF IDF full-text search database.
SPEAKER_01: You might not want to do that either.
SPEAKER_01: So what we do is we break up into sections, smaller sections.
SPEAKER_01: And there's a naive way to do that, which is just every N
SPEAKER_01: characters you chop and then you create a new section.
SPEAKER_01: And that's obviously not ideal
SPEAKER_01: because you lose very relevant context.
SPEAKER_01: And again, these are legal documents.
SPEAKER_01: So they're well-structured typically,
SPEAKER_01: as long as it's not an image.
SPEAKER_01: And so they come in sections and then subsections and bullets.
SPEAKER_01: And so what we try and do is our pipeline
SPEAKER_01: semantically chunk things into essentially sensible chunks
SPEAKER_01: that cut at normal places.
SPEAKER_01: And then we still retain the hierarchy
SPEAKER_01: of where that chunk came from
SPEAKER_01: so that we can reproduce it later.
SPEAKER_01: And then we also store metadata, of course,
SPEAKER_01: and things about where this document came from.
SPEAKER_01: They're like the root document.
SPEAKER_01: And then eventually we have these text chunks
SPEAKER_01: and we embed those both dense and sparse
SPEAKER_01: and we store them in a vector database.
SPEAKER_01: And that's eventually what we'll then query over
SPEAKER_01: when we go to actually make a determination.
SPEAKER_01: But I think it is, I could probably talk for the next 60 minutes
SPEAKER_01: about kind of this process of chunking.
SPEAKER_01: I think we spent a lot of time there
SPEAKER_01: and it's a very important part of this process.
SPEAKER_01: I think if you do a naive implementation,
SPEAKER_01: you leave a lot of accuracy on the table essentially.
SPEAKER_01: Yeah, wouldn't mind having you dig into some of the work
SPEAKER_01: that you've done to kind of assess the lift
SPEAKER_01: on the semantic chunking.
SPEAKER_01: And what you've seen there, I think, as you alluded to,
SPEAKER_01: like a lot of folks will pull a rag library off the shelf
SPEAKER_01: and it'll give you three or four ways
SPEAKER_01: to chunk number of characters and whatnot.
SPEAKER_01: And folks will do one of that, but there's often ways
SPEAKER_01: to take advantage of the inherent structure
SPEAKER_01: and the information that you're trying to capture.
SPEAKER_01: How did you approach that?
SPEAKER_01: Was it just obvious that, hey, we're going
SPEAKER_01: to do this based on sections because it's a legal document
SPEAKER_01: right there?
SPEAKER_01: Or did you iterate on that for a while?
SPEAKER_01: I think it was obvious that when you look at one
SPEAKER_01: of these documents as a human, it's very obvious how
SPEAKER_01: if you were going to break it up, how you would like to break it up.
SPEAKER_01: And so it was clear, I guess, what should happen?
SPEAKER_01: The question is, how do you make that happen in a way?
SPEAKER_01: Doing it for one document is easy.
SPEAKER_01: Doing it for the millions of different documents
SPEAKER_01: that we've pulled in a way that's generalizable
SPEAKER_01: is much more difficult.
SPEAKER_01: And so I guess the details there go into,
SPEAKER_01: as we ingest these documents, we have
SPEAKER_01: a number of different buckets.
SPEAKER_01: You could call them of different structures
SPEAKER_01: of legal documents that we have parsers.
SPEAKER_01: And a lot of these are LLM backed parsers,
SPEAKER_01: but like bespoke parsers that for that particular type
SPEAKER_01: of document, so we will either from the metadata of the document
SPEAKER_01: or through an LLM call, determine, hey, what
SPEAKER_01: is this kind of document?
SPEAKER_01: Is it a case law?
SPEAKER_01: Because a ruling from a judge will look different than the legislation,
SPEAKER_01: the statute law, which will look different from a bulletin
SPEAKER_01: or a notice that the Department of Revenue releases.
SPEAKER_01: And so we have bespoke parsers for each of those.
SPEAKER_01: And some of those, yeah, they involve LLM tool calls.
SPEAKER_01: Some of them are fully just algorithmic
SPEAKER_01: because the structure is all there and it works fine enough.
SPEAKER_01: But I think that's where the devil is in the details on it.
SPEAKER_01: It's a human you look at it exactly what to do
SPEAKER_01: on each of these different examples,
SPEAKER_01: but how do you do it in a way where
SPEAKER_01: it's generalizable across languages, across jurisdictions?
SPEAKER_01: Can you dig into a little bit more detail
SPEAKER_01: on the dense versus sparse aspect of what you're doing?
SPEAKER_01: We started with just a dense representation,
SPEAKER_01: which felt correct at the time.
SPEAKER_01: And if we think about the query that will eventually
SPEAKER_01: run, if we're looking for relevant passages about SAS,
SPEAKER_01: every jurisdiction has a different,
SPEAKER_01: ignoring even different languages, of course.
SPEAKER_01: But every jurisdiction, even in English,
SPEAKER_01: might have a slightly different way
SPEAKER_01: that they describe SAS.
SPEAKER_01: And especially in legislation, legislation
SPEAKER_01: reads very old, their description of SAS
SPEAKER_01: will be very antiquated.
SPEAKER_01: It might even talk about CDs and things of this nature.
SPEAKER_01: And so from the beginning, I think it was a fair assumption.
SPEAKER_01: My opinion was that we should be using a dense embedding
SPEAKER_01: that semantically embed these passages.
SPEAKER_01: And so that is what we started with.
SPEAKER_01: I think what we found, and when we brought sparse back into it,
SPEAKER_01: was there are times, especially when
SPEAKER_01: it comes to citations and pulling out certain terms
SPEAKER_01: from passages that come from the dense embeddings,
SPEAKER_01: where you also want to search sparse,
SPEAKER_01: where you want to do a full text search of certain words
SPEAKER_01: and certain terms and pull those in as well,
SPEAKER_01: and then compare the two of them.
SPEAKER_01: And what we saw was a pretty good increase in accuracy
SPEAKER_01: on the citation side.
SPEAKER_01: So we have some evals that we run on the retrieval part
SPEAKER_01: that we have a baseline of these citations,
SPEAKER_01: or at these passages are the ones that
SPEAKER_01: should be retrieved for these queries.
SPEAKER_01: And as we kind of layered sparse back into that,
SPEAKER_01: we saw another increase in accuracy.
SPEAKER_01: And so that's kind of what we stuck with.
SPEAKER_01: By when you refer to dense and sparse,
SPEAKER_01: are you, it sounds like you're talking
SPEAKER_01: about embeddings versus full text search,
SPEAKER_01: as opposed to like two tiers of embeddings
SPEAKER_01: or something like that.
SPEAKER_01: Yeah, that's right.
SPEAKER_01: So yeah, dense is definitely embeddings.
SPEAKER_01: Like, yeah, semantic embeddings
SPEAKER_01: that we use open AI is embedding models for.
SPEAKER_01: And then, yeah, when I say sparse,
SPEAKER_01: I'm referring to, in our case,
SPEAKER_01: we use pine cone to essentially create a sparse representation.
SPEAKER_01: So we've loaded a vocabulary
SPEAKER_01: and then each passage is fed through
SPEAKER_01: and it keeps an index like full text search
SPEAKER_01: of the different terms and their different usages
SPEAKER_01: across passages.
SPEAKER_01: So it's not quite, you know, elastic search
SPEAKER_01: or Apache Lucene, but it's sparse in like a TF IDF type
SPEAKER_01: implementation.
SPEAKER_01: Got it.
SPEAKER_01: So you've got a predefined vocabulary
SPEAKER_01: and as you pass these documents in the pine cone,
SPEAKER_01: it's just flagging which documents
SPEAKER_01: talk about which of these terms?
SPEAKER_01: Yeah, which passages are talking about which terms.
SPEAKER_01: So then you can search, yeah.
SPEAKER_01: So then when, then you can search over them
SPEAKER_01: and find, you know, if this one uses this particular term
SPEAKER_01: that's not frequently used across the corpus,
SPEAKER_01: it'll be a high result.
SPEAKER_01: And where are the search terms coming from?
SPEAKER_01: Like, do you, you know, is it,
SPEAKER_01: I'm kind of getting ahead of the here answer,
SPEAKER_01: but I'm imagining like document comes in
SPEAKER_01: and the first pass is to see if it's an all relevant
SPEAKER_01: to the task in hand.
SPEAKER_01: And so you're just searching for a bunch of terms
SPEAKER_01: to screen the document, is that the idea?
SPEAKER_01: Yeah, not quite, but I think, yeah, it's a great question.
SPEAKER_01: So the query comes from something slightly upstream,
SPEAKER_01: which we also use T-RAM for,
SPEAKER_01: and that is kind of the first step
SPEAKER_01: for us to support a particular product
SPEAKER_01: is for us to create a, what we call a taxonomy
SPEAKER_01: of that product.
SPEAKER_01: So what that means is we create kind of like a tree structure
SPEAKER_01: of for this particular product type,
SPEAKER_01: what are the different characteristics
SPEAKER_01: across the world that affect its taxability?
SPEAKER_01: So an example might be for clothing.
SPEAKER_01: Clothing that is made for children versus made for adults
SPEAKER_01: can have different tax abilities.
SPEAKER_01: And you can imagine more questions like this
SPEAKER_01: where, you know, maybe pants have a different taxability
SPEAKER_01: than shirts, something like that.
SPEAKER_01: And we build this big tree.
SPEAKER_01: And what the query that eventually gets fed into T-RAM
SPEAKER_01: is essentially a, well, it's a couple of things,
SPEAKER_01: but one thing is a description of that particular type
SPEAKER_01: of product.
SPEAKER_01: So in our clothing example, maybe it's adult pants.
SPEAKER_01: And so we have a description of adult pants.
SPEAKER_01: And so that is the main query that we put into the system
SPEAKER_01: to then pull out relevant passages.
SPEAKER_01: And we'll use filtering, of course.
SPEAKER_01: You know, if we only, we're doing determinations for Florida,
SPEAKER_01: we'll filter to only the passages that come from Florida's
SPEAKER_01: corpus of tax law.
SPEAKER_01: And then we're just looking for relevant portions
SPEAKER_01: to this particular product type,
SPEAKER_01: which we have an LLM generated
SPEAKER_01: few sentence description of.
SPEAKER_01: Got it.
SPEAKER_01: And so this is, this query is kind of,
SPEAKER_01: I guess I'm trying to place this query in the context
SPEAKER_01: of like a document being fed through
SPEAKER_01: an ingestion pipeline, an ingestion pipeline.
SPEAKER_01: And this is maybe after the pipeline,
SPEAKER_01: you've got this retrieval system.
SPEAKER_01: And now you're trying to use this retrieval system
SPEAKER_01: to update the deterministic model, for example.
SPEAKER_01: Is that the right way to think about it?
SPEAKER_01: Yeah, so we've built this big index of law, right?
SPEAKER_01: The tax law from every jurisdiction.
SPEAKER_01: And then a query comes in, which is a description of a product
SPEAKER_01: with a little other information around it.
SPEAKER_01: And then we want to find all the relevant passages
SPEAKER_01: in that jurisdiction for that product.
SPEAKER_01: So yeah, the index itself is just all of the legislative data.
SPEAKER_01: And then the query is a particular type of search
SPEAKER_01: we want to run to pull relevant pieces of legislation.
SPEAKER_01: So you've built this ingestion pipeline
SPEAKER_01: and this retrieval system, it immediately calls to mind
SPEAKER_01: the R and rag.
SPEAKER_01: And it may be that's what you're not doing
SPEAKER_01: ultimately is generation.
SPEAKER_01: But certainly the idea of like taking a bunch of context
SPEAKER_01: and sticking it into an LLM and having the LLM do the thing,
SPEAKER_01: something that you think about,
SPEAKER_01: what's your take on the whole like rag is dead,
SPEAKER_01: retrieval is dead, argument that some folks make.
SPEAKER_01: Yeah, I think, yeah, I was thinking about this this morning.
SPEAKER_01: I think for some use cases, it's certainly true.
SPEAKER_01: And I think we could set up some sort of system
SPEAKER_01: where we just have all this legislation in a file system
SPEAKER_01: and then an agent can grapple over it
SPEAKER_01: and find the relevant pieces that way.
SPEAKER_01: And I think, yeah, for some sorts of problems that works well,
SPEAKER_01: I think for us, or at least for this particular problem,
SPEAKER_01: because we are so sensitive to accuracy
SPEAKER_01: and we're so sensitive to the exact right citation,
SPEAKER_01: essentially we need like a more finely tuned scalpel
SPEAKER_01: to like find us the relevant portion
SPEAKER_01: and we need it to be highly accurate.
SPEAKER_01: And so anecdotally, at least when I use cloud code or something
SPEAKER_01: and I see it gripping through the code base,
SPEAKER_01: there's lots of times it misses.
SPEAKER_01: Like I'll go off and I'll find a file
SPEAKER_01: that I really wish it would have found.
SPEAKER_01: Like this file had the answer I was looking for.
SPEAKER_01: And so maybe we're on a path, five years from now,
SPEAKER_01: our rag system's still working the way they are today.
SPEAKER_01: I'm sure they won't be.
SPEAKER_01: But as of today, I don't think agents
SPEAKER_01: are just searching over the file system
SPEAKER_01: grepping over it.
SPEAKER_01: Is that a point where we could switch over and not lose accuracy?
SPEAKER_01: Talk a little bit about the citations
SPEAKER_01: that you mentioned, how you use those
SPEAKER_01: and how the retrieval system helps you deliver them.
SPEAKER_01: As part of the ingestion process,
SPEAKER_01: we carry through a hierarchy of these different passages
SPEAKER_01: of text that we end up indexing
SPEAKER_01: and each of them carries a citation.
SPEAKER_01: And different passages might share the same citation,
SPEAKER_01: but that's very important for us eventually upstream
SPEAKER_01: when the tax expert goes to review.
SPEAKER_01: Because those citations also have links,
SPEAKER_01: which will link the tax expert out
SPEAKER_01: to the source document where we collected this.
SPEAKER_01: Because a lot of times they want to review
SPEAKER_01: that the tax expert that is, they want to review,
SPEAKER_01: a bit more context than maybe the model gave them
SPEAKER_01: in its breakdown of the citation.
SPEAKER_01: Because the citation, the model will verbatim
SPEAKER_01: give some of the citation back and then a bit of reasoning,
SPEAKER_01: but sometimes they want to expand on it.
SPEAKER_01: And so they'll click out and read the citation.
SPEAKER_01: But essentially, the way that we've handled citations
SPEAKER_01: is through this hierarchy and tagging
SPEAKER_01: of passages of which citation they came from,
SPEAKER_01: which, again, I think in theory sounds easy,
SPEAKER_01: but there's a process at the beginning
SPEAKER_01: with those parsers I mentioned earlier
SPEAKER_01: to make sure that we're pulling the actual correct
SPEAKER_01: like legal citation.
SPEAKER_01: You also experimented with using fine tuning,
SPEAKER_01: RFT in particular for your process.
SPEAKER_01: Can you talk a little bit about where it fits in?
SPEAKER_01: Yeah, so we saw a big jump with O1,
SPEAKER_01: OpenAI's O1 that came out in December of 24, I believe.
SPEAKER_01: Yeah, the first reason model pretty much right out of the gate.
SPEAKER_01: We swapped out the model names like everyone does
SPEAKER_01: and we tried out this new model.
SPEAKER_01: Which task in your pipeline in particular?
SPEAKER_01: Yeah, this final task of like given a certain product type,
SPEAKER_01: deter and jurisdiction, determine its taxability in that region,
SPEAKER_01: which is what the tax expert themselves review.
SPEAKER_01: So yeah, we have eVals, even then we had eVals that would run.
SPEAKER_01: So we plugged it in, it did quite well.
SPEAKER_01: We ran some through and we're impressed.
SPEAKER_01: So we were already like, we're on board with reasoning models.
SPEAKER_01: It was clear that like our use case was well suited
SPEAKER_01: to that extra thinking or those extra tokens
SPEAKER_01: that are spent considering the prompt
SPEAKER_01: and what the answer might be.
SPEAKER_01: And so we're excited when OpenAI reached out
SPEAKER_01: to us to be a part of their alpha program
SPEAKER_01: for reinforcement fine tuning, which is essentially
SPEAKER_01: fine tuning on their reasoning models.
SPEAKER_01: And what we use, you know, with any fine tuning,
SPEAKER_01: you need to provide examples essentially in like standard SFT.
SPEAKER_01: And in RFT, you need to provide that as well
SPEAKER_01: and then you need to provide a greater.
SPEAKER_01: And what we had that was very useful was feedback
SPEAKER_01: from our human tax experts.
SPEAKER_01: Every time the model T-RAM had gotten something wrong
SPEAKER_01: on a determination.
SPEAKER_01: So as the tax experts are reviewing,
SPEAKER_01: when the model is incorrect, they leave feedback.
SPEAKER_01: And they give that feedback
SPEAKER_01: similar to how they would give feedback to like a colleague
SPEAKER_01: who had maybe a more junior colleague
SPEAKER_01: that had made this determined.
SPEAKER_01: Text blurb about what they thought.
SPEAKER_01: And explain it, an explanation in a way
SPEAKER_01: where you want that person to get better
SPEAKER_01: and you want that person to have this, you know,
SPEAKER_01: extra context that maybe isn't clear
SPEAKER_01: from just the legislation.
SPEAKER_01: So some, you know, background information
SPEAKER_01: about how Alabama treats a certain vocabulary word,
SPEAKER_01: something like that.
SPEAKER_01: And what we found was that was a very,
SPEAKER_01: so I guess twofold, we had already like a set of questions
SPEAKER_01: that we knew the model struggled with today
SPEAKER_01: because it had missed them.
SPEAKER_01: And then we had a way to give really great signal
SPEAKER_01: through the feedback and through the fact
SPEAKER_01: that of course we had the correct answer.
SPEAKER_01: Like they, the tax experts fixed the issue, of course.
SPEAKER_01: And then they also leave the feedback.
SPEAKER_01: So we had the ground truth, we had signal,
SPEAKER_01: and we knew that these were hard problems
SPEAKER_01: that the model had missed previously.
SPEAKER_01: And so that was a really good recipe for RFT.
SPEAKER_01: And we saw improvements with,
SPEAKER_01: during the alpha program with OpenAI on RFT.
SPEAKER_01: And that's what we use in production today
SPEAKER_01: while a different model that we've worked with them to RFT.
SPEAKER_01: But we've seen performance or accuracy improvements.
SPEAKER_01: And that really is the key for us is accuracy.
SPEAKER_01: We track it very closely.
SPEAKER_01: I'm always checking in on it.
SPEAKER_01: We want to know how accurate is the model being
SPEAKER_01: and accurate means, you know,
SPEAKER_01: how often is the tax expert having to make an adjustment
SPEAKER_01: to the model's work.
SPEAKER_01: And I'm curious to your experience with like,
SPEAKER_01: I guess what I call undocumented model changes.
SPEAKER_01: Like, you know, I think you mentioned either
SPEAKER_01: before we started recording or as we've been talking,
SPEAKER_01: your use of cloud code, like, you know,
SPEAKER_01: we've seen anthropic document, you know,
SPEAKER_01: some things they do behind the scenes,
SPEAKER_01: you know, tweaking various things
SPEAKER_01: that change the model performance.
SPEAKER_01: Like do you see a lot of that, you know,
SPEAKER_01: with the, with the models that you use,
SPEAKER_01: like needing, you know, just an inexplicable,
SPEAKER_01: unexplained change in behavior that you need to run down?
SPEAKER_01: I think we see that a lot during model generation changes.
SPEAKER_01: So like we work to not fully rewrite,
SPEAKER_01: but rewrite significantly a lot of our prompts
SPEAKER_01: from model generation change to change.
SPEAKER_01: I think, you know, the things that anthropic
SPEAKER_01: gets up to on cloud code as far as, you know,
SPEAKER_01: sending your query to a quantized model
SPEAKER_01: because, you know, they, they're high traffic,
SPEAKER_01: I guess they would never admit to something like that,
SPEAKER_01: but from the outside, that, that looks like what they're doing.
SPEAKER_01: I think on the API side, because we, you know,
SPEAKER_01: we're using APIs with, with OpenAI,
SPEAKER_01: I think those sorts of changes are less likely
SPEAKER_01: and also would have even bigger backlash.
SPEAKER_01: So we don't, I haven't seen anything, you know,
SPEAKER_01: in intramodel generation,
SPEAKER_01: but certainly every time the model changes,
SPEAKER_01: you know, things change outside.
SPEAKER_01: We can't just simply plug into the new version
SPEAKER_01: and get the best results immediately.
SPEAKER_01: Is there anything in particular you've learned
SPEAKER_01: or specific to your product
SPEAKER_01: with regards to the way you approach evals?
SPEAKER_01: You know, beyond kind of collecting a data set
SPEAKER_01: where, you know, the models had errors in the past
SPEAKER_01: and, you know, running the models through those
SPEAKER_01: or that kind of thing.
SPEAKER_01: I think, yeah, I think that's been the main thing
SPEAKER_01: and I think because, yeah,
SPEAKER_01: because we have these human experts,
SPEAKER_01: maybe the one part that's not as standard is,
SPEAKER_01: you know, because the tax experts are reviewing these things,
SPEAKER_01: we have an ever-growing list of evals
SPEAKER_01: because it's very easy for the experts.
SPEAKER_01: There's a toggle essentially they can click that says,
SPEAKER_01: like, hey, this is a difficult one,
SPEAKER_01: you should include it in the eval set
SPEAKER_01: and they give a description of why.
SPEAKER_01: So we have this, like, growing list of evals
SPEAKER_01: that we can pull from,
SPEAKER_01: which I think is important for the model,
SPEAKER_01: especially because we do this RFT with OpenAI.
SPEAKER_01: You know, I've even done this for a while,
SPEAKER_01: but I think if we went back and ran the evals
SPEAKER_01: on like our original evals that were running,
SPEAKER_01: you know, a year and a half ago,
SPEAKER_01: it would not be nearly as useful
SPEAKER_01: as the evals that are running today
SPEAKER_01: because the model has changed and improved
SPEAKER_01: and maybe degraded actually in some particular ways.
SPEAKER_01: Hey, going back to retrieval,
SPEAKER_01: you had a previously discussed
SPEAKER_01: like some interesting things you're doing around
SPEAKER_01: reordering and expanding and kind of using an LLM
SPEAKER_01: in the retrieval process to enhance your results.
SPEAKER_01: And I don't think we dug into that.
SPEAKER_01: Can you elaborate on that a little bit?
SPEAKER_01: Yeah, so that would be downstream from, you know,
SPEAKER_01: we've built this index of all the legislative data
SPEAKER_01: like we've talked about.
SPEAKER_01: And then when a query comes in,
SPEAKER_01: we have a multi-step process to essentially build up
SPEAKER_01: the relevant context for that query
SPEAKER_01: before we eventually send it off
SPEAKER_01: to like the final reasoning model
SPEAKER_01: to reason through the actual like taxability
SPEAKER_01: of the product.
SPEAKER_01: And so what that looks like is an initial search
SPEAKER_01: into our database, of course,
SPEAKER_01: as far as I'm dense to pull out relevant passages.
SPEAKER_01: We then use LLM as a judge or LLM as a re-ranker
SPEAKER_01: to re-rank those into more relevant pieces.
SPEAKER_01: We then expand each of the passages
SPEAKER_01: because we've retained the hierarchical nature of them.
SPEAKER_01: So we can grab, you know, the previous
SPEAKER_01: and the following chunks or passages
SPEAKER_01: and build out the context of the relevant passages
SPEAKER_01: and then we'll give that back to an LLM again
SPEAKER_01: to then reorder and potentially throw away certain things
SPEAKER_01: that now seem like they're not relevant
SPEAKER_01: as we've added context.
SPEAKER_01: And we repeat this process until we hit either a certain amount
SPEAKER_01: of length or certain confidence
SPEAKER_01: that we have the relevant context.
SPEAKER_01: And then that goes off to like the final step,
SPEAKER_01: the LLM to make the actual determination.
SPEAKER_01: But that was a change made a little bit later
SPEAKER_01: in the process as well that, you know,
SPEAKER_01: in the search for accuracy, increasing accuracy,
SPEAKER_01: another wrinkle that I think added quite a bit.
SPEAKER_01: You continue until you reach a certain level
SPEAKER_01: of confidence.
SPEAKER_01: Is that based on an LLM as judge type of scenario,
SPEAKER_01: but I can LLM's determination of confidence?
SPEAKER_01: Yep, that's right.
SPEAKER_01: And that's basically by looking back at the previous,
SPEAKER_01: like we'll give it both the previous passages
SPEAKER_01: that were fed in on the last pass before they were expanded.
SPEAKER_01: And then the current ones as well,
SPEAKER_01: because at some point, you know, you've expanded too far
SPEAKER_01: and now the legislation is talking about automobiles
SPEAKER_01: or something that's no longer relevant.
SPEAKER_01: So you're just asking if there's been a scope change
SPEAKER_01: or something like that essentially?
SPEAKER_01: Yeah, is the added context actually useful?
SPEAKER_01: Like is it on target for what we're looking for?
SPEAKER_01: And then, you know, where do you in this search for increase
SPEAKER_01: accuracy?
SPEAKER_01: Where do you see like your next jump coming from?
SPEAKER_01: Yeah, part of it is model providers.
SPEAKER_01: It's great every time, you know, the release cadence
SPEAKER_01: has been even faster from OpenAI anthropics.
SPEAKER_01: So that's been great.
SPEAKER_01: We see a bump once we adjust things
SPEAKER_01: with every model that they release.
SPEAKER_01: I think it's further as far as like further refinement
SPEAKER_01: of the RFT process with OpenAI.
SPEAKER_01: I think that's kind of part,
SPEAKER_01: that's a big part of the way we'll get to, you know,
SPEAKER_01: where we aim to get.
SPEAKER_01: And what we want is, like I mentioned,
SPEAKER_01: a human expert reviews every one of these determinations today.
SPEAKER_01: They go through every single one.
SPEAKER_01: And you know, right now that takes them around 10 seconds,
SPEAKER_01: nine seconds to review each of them on average.
SPEAKER_01: So that's incredibly fast compared to the incumbents
SPEAKER_01: who are doing it totally manually.
SPEAKER_01: But we'd like to increase that even further.
SPEAKER_01: And one way to do that, the best way to do that
SPEAKER_01: is if they could take a random sampling instead.
SPEAKER_01: So if we can get our accuracy to a point
SPEAKER_01: where we're confident that given a random sample
SPEAKER_01: of some number from the determinations the model is done,
SPEAKER_01: if those are accurate, we don't need to review
SPEAKER_01: every single one of the determinations.
SPEAKER_01: So that's kind of the North Star,
SPEAKER_01: at least on this front that we're marching towards.
SPEAKER_01: And I think RFT will be a big part of that
SPEAKER_01: because chasing this long tail,
SPEAKER_01: right, chasing the nines of accuracy.
SPEAKER_01: A lot of it starts to become very,
SPEAKER_01: to get the correct answer,
SPEAKER_01: it's very sales tax focused, right?
SPEAKER_01: You need to have a really deep understanding of tax law.
SPEAKER_01: It's deeper than like these models have
SPEAKER_01: just like out of the box based on their training data.
SPEAKER_01: And so I think that is, you know,
SPEAKER_01: we'll make changes to our retrieval process of course
SPEAKER_01: and then those will be somewhat helpful.
SPEAKER_01: But I think to get those last couple of accuracy points
SPEAKER_01: that we need, it'll be working with, you know,
SPEAKER_01: the frontier labs to try and do something more bespoke.
SPEAKER_01: And I asked previously about kind of this, you know,
SPEAKER_01: rag is dead question.
SPEAKER_01: But I'm wondering the degree to which context length
SPEAKER_01: changes the way you approach the problem.
SPEAKER_01: Like it could be that these documents are so structured
SPEAKER_01: a section is gonna be, you know, three to five pages
SPEAKER_01: and it doesn't really matter if you have access
SPEAKER_01: to a two million token, you know, context window
SPEAKER_01: or it could be that, you know,
SPEAKER_01: there are other ways you can use that context.
SPEAKER_01: How do you think about the impact of context window?
SPEAKER_01: I think that was actually one of the big reasons
SPEAKER_01: why we saw jump with the release of 01 back in the day
SPEAKER_01: was I think reasoning models are much more capable
SPEAKER_01: of reasoning over their full context.
SPEAKER_01: Whereas non-reasoning models, yeah,
SPEAKER_01: you got real degradation as even if it supported,
SPEAKER_01: you know, 128 K tokens, when you push that limit,
SPEAKER_01: it was not, you know, needle in the haystack
SPEAKER_01: wasn't great on those sorts of things.
SPEAKER_01: And so I think we saw big improvements there
SPEAKER_01: with reasoning models.
SPEAKER_01: And so it's still a balance for us.
SPEAKER_01: Like we, like I kind of mentioned earlier,
SPEAKER_01: we don't need to fill up and we don't fill up
SPEAKER_01: the context window to its max.
SPEAKER_01: But a big unlock was models where we could give it more,
SPEAKER_01: where maybe we could be a little less precise
SPEAKER_01: on the retrieval portion and expand,
SPEAKER_01: expand these passages a little more aggressively.
SPEAKER_01: I think before when context was more limited,
SPEAKER_01: you know, we were being very selective
SPEAKER_01: on which passages we're feeding in
SPEAKER_01: because we, you know, we only had so much we could give it
SPEAKER_01: before the model just kind of withdraw its hands up.
SPEAKER_01: And so that, that was a big unlock.
SPEAKER_01: So, yeah, we don't push the boundary right on the edge,
SPEAKER_01: but I think as reasoning models improve,
SPEAKER_01: as the context window gets bigger,
SPEAKER_01: again, we won't fill it up all the way,
SPEAKER_01: but that's a good sign that the model can handle
SPEAKER_01: more, more tokens than we're giving it today.
SPEAKER_01: And that means we can be less precise
SPEAKER_01: a bit on the retrieval portion
SPEAKER_01: and still get the results that we're looking for.
SPEAKER_01: How much time do you spend thinking about trying
SPEAKER_01: to reduce token costs,
SPEAKER_01: either by kind of refactoring from larger models
SPEAKER_01: to smaller models or via other methods?
SPEAKER_01: LLMs compared to lawyers,
SPEAKER_01: like human tax lawyers are considerably cheaper,
SPEAKER_01: even the most expensive LLMs.
SPEAKER_01: So yeah, this isn't, and this isn't also something,
SPEAKER_01: this isn't a process where, you know,
SPEAKER_01: we're pushing through billions of tokens that we...
SPEAKER_01: I guess that helps that you're building
SPEAKER_01: a deterministic system
SPEAKER_01: and that is the thing that's, you know,
SPEAKER_01: the kind of the inline, online system
SPEAKER_01: as opposed to an LLL inference call.
SPEAKER_01: Yeah, exactly.
SPEAKER_01: We're not cost sensitive
SPEAKER_01: and that also means we're not latency sensitive either.
SPEAKER_01: So it's very nice, those are two things
SPEAKER_01: that we don't even really have to consider very closely.
SPEAKER_01: Quite luxuries, right?
SPEAKER_01: Yeah, nice, nice.
SPEAKER_01: Maybe to kind of wrap things up,
SPEAKER_01: where do you see things going for, you know, both T-RAM
SPEAKER_01: and kind of AI and feels like tax more broadly?
SPEAKER_01: Yeah, I think we have a clear path on T-RAM,
SPEAKER_01: kind of what I mentioned earlier of deep, you know,
SPEAKER_01: increasing accuracy and decreasing human time
SPEAKER_01: spent reviewing.
SPEAKER_01: So we'll continue to chase those metrics
SPEAKER_01: and improve them and that will allow us to be even more accurate
SPEAKER_01: and even more nimble and cover more jurisdictions
SPEAKER_01: in the world.
SPEAKER_01: So that's certainly somewhere we're going to keep pushing.
SPEAKER_01: Then there's other parts of this
SPEAKER_01: that, for example, one thing we talked about
SPEAKER_01: was these taxonomies that we build.
SPEAKER_01: That, you know, identify the different characteristics
SPEAKER_01: of a product that impact their taxability across the world.
SPEAKER_01: Currently we do that with our human experts
SPEAKER_01: because this is something,
SPEAKER_01: it doesn't need to be repeated for every jurisdiction.
SPEAKER_01: If this is like a one time thing that we, you know,
SPEAKER_01: we create this taxonomy just for SAS or just for clothing.
SPEAKER_01: So today we're doing that the traditional way
SPEAKER_01: with human experts.
SPEAKER_01: But if you think about what they're doing
SPEAKER_01: and what the question is there,
SPEAKER_01: we have all the data sitting in our index
SPEAKER_01: to build these taxonomies, right?
SPEAKER_01: For every jurisdiction, we know inherently
SPEAKER_01: in that data somewhere holds the answer
SPEAKER_01: to how to, you know, what are the different characteristics
SPEAKER_01: that affect taxability.
SPEAKER_01: And so I think that's another obvious spot
SPEAKER_01: that would also allow us to move even more quickly,
SPEAKER_01: add more product types.
SPEAKER_01: There's, you know, we'd like to increase the accuracy
SPEAKER_01: and the frequency of these ongoing scrapes
SPEAKER_01: that we're doing as you can probably imagine.
SPEAKER_01: There's a huge amount of data sources
SPEAKER_01: that we're looking at right now
SPEAKER_01: and, you know, not all of them can be scraped immediately
SPEAKER_01: or every hour or whatever.
SPEAKER_01: So we'd like to increase that and increase accuracy
SPEAKER_01: of the outcomes of what those changes do in our system.
SPEAKER_01: And then there's some tangential things around like,
SPEAKER_01: you know, we'd like to make it as easy as possible
SPEAKER_01: for customers to move from a different tax solution
SPEAKER_01: to sphere.
SPEAKER_01: And one way to, you know, a big reason
SPEAKER_01: people don't switch tax solutions
SPEAKER_01: or why they become entrenched is because they've spent
SPEAKER_01: so much effort in mapping their products
SPEAKER_01: to tax codes for a particular system.
SPEAKER_01: And what we're preliminarily doing with T-RAM
SPEAKER_01: is an automatic mapping from, you know,
SPEAKER_01: some competitors' tax codes
SPEAKER_01: or really any classification system.
SPEAKER_01: So if you've classified your products using HS codes,
SPEAKER_01: for example, which is what is used for tariffs,
SPEAKER_01: we could take in any product classification
SPEAKER_01: and map that to a sphere tax code
SPEAKER_01: and then the switching cost to switch to sphere
SPEAKER_01: is just seriously lowered
SPEAKER_01: and you can actually get people to consider making the switch.
SPEAKER_01: So I think there's, you know, we haven't talked about
SPEAKER_01: Ian Boiseing and there's lots of other things.
SPEAKER_01: But at the end of the day,
SPEAKER_01: it all stems from having this index of legislation
SPEAKER_01: across the world set up so that we can query over it.
SPEAKER_01: Out of curiosity, what are the tools that you use
SPEAKER_01: and think of as like your biggest AI unlock
SPEAKER_01: from a personal workflow perspective?
SPEAKER_01: Yeah.
SPEAKER_01: So I've been a subscriber to ChatGBT for a long time,
SPEAKER_01: you know, option space on my Mac.
SPEAKER_01: I use it all the time.
SPEAKER_01: Cloud code, I have that pulled up, you know, all day,
SPEAKER_01: every day that's been a massive unlock for us
SPEAKER_01: while for me personally
SPEAKER_01: and I think across the engineering team here at Sphere
SPEAKER_01: and we were also beginning work on something akin
SPEAKER_01: to like strike minions.
SPEAKER_01: So strike put out a paper with something they called minions
SPEAKER_01: which are like AI agents that are running around
SPEAKER_01: and looking at the code base and opening up PRs
SPEAKER_01: and working together to kind of improve
SPEAKER_01: taking care of things like DependaBot, PRs that get open
SPEAKER_01: and so that's something we're looking at as well to build out.
SPEAKER_01: How can we do that in a Sphere-specific way
SPEAKER_01: and kind of related to that?
SPEAKER_01: Also, what other tools can we add to our internal AI agents?
SPEAKER_01: What skills can we add to make them even more valuable
SPEAKER_01: for us based on our particular use case?
SPEAKER_01: You know, where to pull data, where to look for,
SPEAKER_01: you know, these AI agents should be plugged
SPEAKER_01: into T-RAMs internal index
SPEAKER_01: and be able to give answers from the legislation.
SPEAKER_01: So I think there's, you know, that stuff is still mason for us
SPEAKER_01: but yeah, I feel like I'm surrounded by LLM's all day,
SPEAKER_01: every day.
SPEAKER_01: Awesome. Awesome.
SPEAKER_01: Well, Alex, thanks so much for jumping on
SPEAKER_01: and sharing a bit about, you know, what you're up to
SPEAKER_01: this year and how you're using AI.
SPEAKER_01: Thank you, Sam, for having me.
SPEAKER_01: Thank you.