TNO043: Under the Manhole Cover: The Architecture of an Internet Exchange

In an IT world full of abstraction, overlays, and virtualization, it’s important to remember the physical infrastructure that supports all those things. So let’s get inside Mass IX, the Massachusetts Internet Exchange, to get a holistic view of the logical architecture and protocol mechanics of peering and Internet exchanges, as well as the iron, steel, and glass of Internet infrastructure that keep all the bits moving. Our guide is James Jun, IP Core Network Architect and Managing Director at TOWARDEX and the Director of Mass IX. 

We talk about how early exposure to Cat3 Ethernet cabling hooked James on networking, the operational and technical challenges of running an IX, and why James is excited about the future of colos.

Episode Link:

TOWARDEX

Episode Transcript:

This episode was transcribed by AI and lightly formatted. We make these transcripts available to help with content accessibility and searchability but we can’t guarantee accuracy. There are likely to be errors and inaccuracies in the transcription. 

 

Scott Robohn (0:06 – 2:18) Welcome to Total Network Operations, the project with a podcast dedicated to hardworking network operators like you who deliver all the packets. We drive conversations on great ideas in NetOps so you can make informed decisions for your network and your team and to spur constructive dialogue between vendors, NetOps teams, and across the whole community. I’m your host, Scott Robohn. On today’s episode, I want to say hi from Boston, Mass., technically Somerville, Mass., or Somerville for the local pronunciation, for this special episode of TNOps on the Road. We have a little bit of a different setup for this episode. I have this weird, unique hobby of taking pics of manhole covers for telco, cable, and other comms infrastructure. It’s hashtag robopipes on LinkedIn, some are on Instagram and Facebook. I started doing this when I was doing a lot of walking to lose weight. I was probably looking down too much while I walked around, but it had this payoff of getting some really good documentation of comms infrastructure. As a networker, I’m fascinated by the physical infrastructure, the whole idea that the internet is built on, these pipes beneath our feet and beneath the streets, so it just made sense to start documenting it. I’m going to be putting more of a focus on physical infrastructure, call it broad shoulders networking or blue collar networking. It’s an important perspective in a world that’s full of abstraction, indirection, virtualization, and software eats the world. This episode is the first of what I hope will be multiple that put attention on infrastructure. To start this series, I have the pleasure of welcoming James Jun today. James wears at least two hats, and he’s going to tell us about them and more. He’s the IP Core Network Architect and Managing Director at TOWARDEX, and he’s the Director of Mass IX, the Massachusetts Internet Exchange. This combo of roles gives him a very holistic view of logical architecture and protocol mechanics of peering and internet exchange, as well as the iron and steel and glass of internet infrastructure that keep all the bits flowing. So James, welcome. Thanks for hosting me here today in one of your facilities at CoreSite here in Somerville, Mass. Did I get the intro right? Tell me your background, tell me a little about yourself.

 

James Jun (2:18 – 2:50) Absolutely. Well, thank you very much for coming by. It’s nice to meet you finally, and definitely glad to be here. So yes, you definitely got the introduction right. So how I got into networking was when I was in junior high, you know, back when I was a kid. So I was in this school district called Acton-Boxborough in Massachusetts, and they’re one of the nice schools, and they were one of the first school districts in Massachusetts to completely wire all of their classrooms with, you know, it initially started with Cat 3, and then they moved on to Cat 5.

 

Scott Robohn (2:50 – 2:51) Cat 3? Wow.

 

James Jun (2:52 – 3:15) So when I was first exposed to it, it was 97, but I believe their first install was like, I don’t know, I think it was like 95 or 96. So it obviously was already set by the time I got into the school, right? But it was fascinating because when I was a kid, you know, you have all these computers, you know, the school had a lot of Macintosh and all that. There’s Ethernet jack in every… 

 

Scott Robohn (3:16) No Wi-Fi. 

 

James Jun (3:17 – 3:36) Yeah, no Wi-Fi back then, but it’s just crazy seeing the ubiquitousness of, you know, this LAN being next to every computer throughout the whole school, and it looks like a phone jack, a little bigger, right? Right. And it’s connected to this always on, you know…

 

Scott Robohn (3:36 – 3:38) RJ-11s don’t fit in the RJ-45.

 

James Jun (3:38 – 6:07) Yeah, it looks similar, so it’s a little fatter. You plug it in, you got the 10 base D, and then like you can just… And school had this T1 line, you know, back then T1 was a big pipe, right? So it was crazy because at the time, like my exposure to the internet at the time as a kid was on parent’s computer, you get on AOL or use a modem, you always had to do this, you know, we just go sign on, that’s how the whole thing started, right? It was very convoluted, very… not only was it slow, it was a very convoluted process, you gotta sign in, you gotta sign out, you know, it was like that, but then it was really fascinating how in any machine you just…that’s plugged into this jack that’s next to every computer, you just open up Netscape and there’s internet, it’s so fast, I was like, oh my god, this is great. So that’s how I was really… because, you know, we’re all nerdy at the end of the… I think if we ask all of ourselves, right, in this industry, at the end of the day, we are doing networking because we’re all nerds. We just… we, you know, we start because we love computers and then, you know, when you start seeing the fact that computers or back in the, I think, like the 70s and 80s, we used to call them terminals, right, where they start interacting with each other over a network, I think network is that next evolution of where you move your…

What piques your interest, right? And when I saw that in junior high, it really opened my eyes up because it really made me get fascinated about… it made me fall in love with the internet. That’s awesome. Right there. Not just… not AppleTalk or any kind of enterprise that were right from there, it made me fall in love with TCP/IP. And I was like, wow. And then from there is where my networking career essentially began. And then I was very lucky and blessed from there, you know, two to three years later, I think it was like 2000, 2001-ish. Yeah, 2001, I think. The school’s ISP, you know, was nice enough to hire me while I was still in high school, right? That’s awesome. Yeah. Initially, it’s like intern, you know, summer intern kind of basis. So that really, you know, obviously, you know, every company has their ups and downs, no matter which large, small. It really opened my eyes up in two different ways. Number one, it’s the first time in someone’s career or life whatsoever, you know, what have you, where you’re seeing the business, the how the sausage is made, the business aspect of, you know, whether your drama from everything from customer support to building non-paying customers and just office politics and everyone just hating on each other or like, you know, you know, the usual…

 

Scott Robohn (6:07 – 6:09) Never happens. Yeah, yeah. You know, you don’t hate on each other. 

 

James Jun (6:09 – 6:39) Yeah, exactly. You know, the usual dirty laundry of any company, right? You get to really see that, you know, in full view as that type of people interaction, it’s what’s keeping the world connected. You really see a different side of not, you know, from being an end user of a well-oiled machine called the internet to one of the businesses that’s making it all happen. It was really fascinating to see. And I’m very grateful for the opportunity that they gave me, you know, when I was, you know, when I was only a kid.

 

Scott Robohn (6:40 – 6:43) What’s the very first thing they had you do? Like, what was your first assignment?

 

James Jun (6:43 – 7:24) The first thing, if I remember correctly, so I was… so, during the interview, so obviously, you know, I started playing with Linux and, you know, FreeBSD. I was a big FreeBSD fan. So, they had me start working on just simple stuff like just, you know, take tickets for DNS requests. And back in those days, internet was a lot more fun back then. You don’t have the automation, this, you know, software, this properware.

Like nowadays, if you’re going to networking, you might also be a programmer nowadays. But back in those days, you know, network was easy and it was just more fun, right? You know, it was more like, it wasn’t so much called DevOps, it’s just like, well, I guess it is.

But, you know, it was more like, you know, shell and sysadmin stuff, right?

 

Scott Robohn (7:24 – 7:31) You were stringing it all together. Like, every circuit turn up was bespoke, was handmade. 

 

James Jun (7:31 – 9:11) Yep, exactly. And it was a lot more hands-on. Yep, you’re 100% right about that. It was… so, there’s a lot more like interaction, so to speak. Less abstraction, less layers, right? So, you know, in some ways, it scales less than where we are today. So, you know, there’s certainly value where we are today in a technology sense, but you can truly appreciate how the sausage is made back in the day. So, yeah, so I started just doing, you know, handling those tickets, right? And as the time wore on over the year, then, you know, I kind of, they had my role expanded where they put me in front of the routers back with, like, I think it was really fascinating when they had me work on their border routers, where we had this idea of three lines from Verizon. Back then, this wasn’t Verizon. Today, when you think of Verizon, we think of a U.S. Semiconductor. They were actually just going through rebranding. They just started the Verizon brand around that time. I might have been just, it was like, around the middle, I don’t know if it was 2000 or 2001, I think it was 2001, something like that, or maybe 2002. So, anyway, so we had, so this was before, you know, Verizon buying MCI. This was, this was an AS701. This was Verizon GNI.net, right? That was a fun, you know, regional tier two network. It was pretty well engineered for what it was. It didn’t have peering or anything, but, you know, they at least had a good selection of transit providers, right? And so we had DS3 lines, multiple of them doing ECMP. And then, you know, when I, but, you know, when I started working with the network team on those circuits, now I was put on, you know, what, you know, having enabled on their border routers. It’s the first time I touched a router that’s not a Cisco 2600, right?

 

Scott Robohn (9:12 – 9:13) But still as an intern, right?

 

James Jun (9:13 – 9:14) As an intern, yeah.

 

Scott Robohn (9:14 – 9:15) That’s pretty cool.

 

James Jun (9:15 – 9:40) I was in my senior year, right? Yeah, awesome. And it was really fascinating, nerdy experience. I really loved that. And, you know, I still stay connected with those folks. It was a company called Merrimack Education Center. And those folks are, you know, they’ve been acquired and, you know, but, you know, the folks are still there and they’re really good people. They gave me opportunity in ways that really defined, you know, what we do today. 

 

Scott Robohn (9:40 – 9:43) So important to remember the people that give you those first opportunities. 

 

James Jun (9:43 – 9:49) We still stay in touch, I believe. Yeah. And, you know, it’s been fantastic. 

 

Scott Robohn (9:49 – 9:51) Did you hire on there eventually, or did you do other things? 

 

James Jun (9:51 – 10:49) Eventually, so I think it was 2003, I wanted to, you know, kind of get out into some more, you know, different things where I wanted to do some consulting or kind of get out to the hosting or, you know, do a little kind of have a change of pace. So in 2003, I left to, you know, kind of start doing some consulting work for other, you know, people as I got. And usually, you know, being fresh out of high school and being young, you’re trying to find customers. So I started subbing in for one of the small IT integrators, right? So where I’m essentially, you know, they just put me as like 1099 contractor, right?

And then I just do a network for them. And that was fun. That opened me. Lots of different experiences. Lots of different. And I love my first job because, you know, for all the upsides and downsides of any workplace, right? You’re working at a service provider. Sure. And as you know, as a lot of listeners in this podcast know, there’s a huge disconnect and difference between service provider and enterprise.

 

Scott Robohn (10:50 – 10:51) Different technologies.

 

James Jun (10:52 – 11:06) Completely different worlds. Completely different worlds, different, not only spending levels. So, you know, going from that to dealing with enterprise customers, it was a unique experience, so to speak. And they quickly started to get a little boring.

 

Scott Robohn (11:07 – 11:11) I see. I’m seeing a theme here. Keep James’ attention.

 

James Jun (11:12 – 13:07) Yeah. So then after that, I started selling a little bit of, you know, to kind of like do things on my own, right? So I started doing a bit of hosting out of data centers. So, you know, I went to this data center in Waltham. It’s, I think it was called Switching Data at the time. Okay. It got acquired by Equinix. Now it’s, now that building’s empty, you know, Equinix closed down. They have a new data center now nearby.

That’s, you know, way, way nicer. So anyhow, so we started doing that. And then fast forward, you know, it was just off and on, just sidekick here and there. Many years later, eventually around 2012 is when I finally decided to start TORWARDEX as a network provider. So I’ve seen, you know, by that time, you know, I really started seeing a lot of the, you know, at that point I’ve really fully moved on to the colo world, right? So I started interacting and dealing with a lot of customers that are coming to colocation. And back then, you know, 2009 to, you know, mid 2010s was a heyday for colocation in Boston. I think it like that in everywhere, right? There was at a time where everyone was talking about cloud, but there was an inertia of people still staying with, you know, their own environment, right? Nowadays, you know, everyone is a cloud and now there’s some talk about, you know, putting some of the workloads back now, which is great. But this was a heyday of colo and Boston and New England is unique in that we have very high energy costs from a data center perspective. So we are a very, you know, ill-equipped market right now in terms of enterprise colo. So the cloud has effectively destroyed colocation in New England. Like, so there are still, believe me, there are still tons of enterprise colo in Boston, right? But when you compare us to say developing markets or big markets, like even a place like Phoenix or whatever, it is a very sad place of what it used to be. 

 

Scott Robohn (13:08) Interesting. 

 

James Jun (13:09 – 14:01) Yeah, because, you know, and I think that the cloud has kind of, it kind of happened at a wrong time where the combination of the cloud and high energy costs kind of work against the data centers here in Boston. Because what happened was, is that once the energy costs started going up and everyone started moving out to cloud, and now people are talking about moving some of the workloads back to the data centers, right?

But they look at like, well, since we already moved out, now does it really need to come back to our offices anymore? Or why don’t we just take the colo in Houston or Phoenix, right? It doesn’t have to be in Boston anymore. So the return to the colo in Boston is, you know, we are seeing some of that and that’s great. I always love that. I’m a huge fan of old school colo. I think that more, those were the fun times where everyone owning, it’s really about the ownership of the IT processes, right? And I’m not, and I think that the cloud today has a value in the ecosystem and the convenience that it provides. But 100% reliance has been, you know, it’s not been very, it’s been quite boring.

 

Scott Robohn (14:01 – 14:14) So just to make sure I’m tracking with you, like the use of SaaS products for everything primarily. And like, you know, I don’t have to install it on my servers. I don’t need to have those servers in my building. It’s just in the cloud somewhere.

 

James Jun (14:14 – 14:40) Yeah. And that mentality for a while has really done, you know, it hurt colocation in many, many places. I mean, you ask any data, they’ll tell you that. But it’s still, there’s a strong demand for colocation. It’s only going up, right? And, you know, obviously Boston having, being a corner of the US and having high energy costs kind of hurt the data centers even more. But the great news is we’re starting to see some of that come back, even in this market now as a, you know, which is great. So I think we’re at a great time.

 

Scott Robohn (14:40 – 14:45) But you were telling me before we got started, you’re keeping pretty busy. So you might, you must be seeing.

 

James Jun (14:45 – 14:51) Yes, we are. We are seeing a lot more activity. Quite frankly, it’s making me feel young again. So I like it. 

 

Scott Robohn (14:52 – 14:56) That’s awesome. Well, so let’s, you know, we’ve got at least these two chunks of what you do.

 

James Jun (14:56 – 15:20) So, yeah, so, so I apologize. I kind of got sidetracked. No, that’s all good. Yeah. So as we, by that time, you know, I got fully interested in colo. That was more where I saw an opportunity, right? There was one company that everyone remembers called INAP or InterNAP, right? So, you know, sadly, they, you know, they went through chapter 11, chapter 22, chapter 33. We call those numbers, you know, the first time you do chapter 11, it doesn’t work out. So you do it again.

 

Scott Robohn (15:20 – 15:22) Right, right, get a redo.

 

James Jun (15:22 – 17:13) Yeah, you do like I said, man. So INAP’s gone for all intents and purposes. And, but what they really do was they had this, they really invented this concept of blended bandwidth at colo, right? Where they’ll take transit or upstream internet connectivity from all these different tier one or large tier two providers, mix it down. And then they had this like, you know, proprietary simulator, whatever they call it. And then they would sell this, what they call optimized, well-connected internet service to their customers. And, you know, it was very popular amongst colo users, right? So over time, you know, INAP kind of lost its way. Just like, you know, every company has their moments, right? And so, but the colocation demand kept going up. So what you have is, bless you, what you have is just different customers coming into this carrier neutral data centers, like the one we’re sitting in today, where they need blended bandwidth connectivity. So INAP provided this blended bandwidth solution in data centers. And they were obviously selling a lot of that in their own colocation providers. So naturally, a lot of data centers like the carrier neutral data centers, like the one we’re sitting in here today, they, you know, obviously INAP is also a competitor, right? So although they’ve been very good at working with each other, you know, you can be competitors that still have good relationships, right?

But there’s a sales friction, natural sales friction between salespeople, right? So that’s where we saw an opportunity. Let’s be an INAP competitor, come in, run. And this is where I really, we really found our calling is that now, you know, we can really be our own service provider in ways that we wanted. And back then, Boston went through a couple of stages in the internet infrastructure. So in the dot-com days, Boston was one of the most well-connected markets in the USA. If we dial the clock back 25 years from now, right? I’m sorry, 25 years back, right?

 

Scott Robohn (17:14 – 17:19) We had the whole, you know, tech corridor, Route 120, you have so many universities. 

 

James Jun (17:20 – 18:37) Exactly. And back in the days of MediaOne and all that. If you just take random, you know, DSL or 2MAC cable broadband that you bought, and you just run Traceroute, right? Most of the traffic is locally switched or locally routed, I’m sorry, locally routed here in Boston. So if you’re going to, you know, some of the connections, I think there was all, you know, these are all bankrupt company names. So if you’re doing Traceroute from, say, MediaOne to someone’s Flashpoint DSL user, it’ll stay in Waltham in Boston. It was really fascinating. The internet was really, you know, Boston used to be very well-connected. And then after the telecom, the bandwidth market went to a total meltdown, a disastrous meltdown. And then after all of that happened, it became a telecom nuclear winter, right? And then started, that’s when the private equity started coming in. That’s when the mergers and acquisitions, all this stuff. So you had a second generation of providers in the CLX. We had some of the, that created some of the best stories of Boston, like Lightower Fiber Network is one of them. But they also had their downsides, is that this second generation of providers that came into Boston had no clue whatsoever on how to run IP networks. It is, oh, well, maybe I’m going to take, I’m sure I’m going to be eating some of this back. I know there’s really good people in Lightower that are really great. I love them. I’m not talking anything bad about them.

 

Scott Robohn (18:37 – 18:37) Sure, sure.

 

James Jun (18:38 – 18:56) They’re no longer with us. I mean, the company got absorbed by Crown Castle by now. But the point being is that it’s, you know, in their sense, they ran a good network on the IP, and they were a fantastic fiber network. But the problem was, I’m comparing it to the old days of Boston, where we’re, for all intents and purposes, we’re practically a tier one market.

 

Scott Robohn (18:57 – 18:57) Sure.

 

James Jun (18:57 – 19:46) We don’t see, everything was being backhauled to New York. It wasn’t just Lightower, it was also Comcast, all the newcomers, every single one. New York City was literally a default gateway for Boston. It was really bad. It was, you know, there was no peering whatsoever up here. And it was really sad to see. So in 2012, when we started TORWARDEX, Boston was still reeling from a lot of that. I always had this inkling that for our customers, we want to fix this, provide the best connectivity that we can, that we can afford. Obviously, it’s easier said than done. When no one does peering, you’re just a small company, no one’s going to peer with you. So, but the easy way you can start is you can start buying transit. Right. That starts to get very expensive, as you know. But, you know, you start with what you can. So we’re not entirely well connected, but we started being well connected in some of the locations that customers, you know, like a lot, right?

 

Scott Robohn (19:46 – 19:46) Sure.

 

James Jun (19:46 – 20:14) Whether it be, you know, you want to like Comcast is probably the first one to start with or you start with local networks. And over time, we got really lucky in that we’re able to build this report and relationships. This is the first time where I really felt the people relationships really matter. Because, you know, us as networkers, we only think about turning up circuits, turning up BGP. Those are like one of the exciting moments in every day in our careers, right? 

 

Scott Robohn (20:14 – 20:20) So be specific. What kind of relationships are you talking about with people in the regulatory structure, with the individual companies?

 

James Jun (20:20 – 20:28) Not even just basic. All of it? The peering coordinators, the product sales teams. Sure. Because a lot of time as a small company. 

 

Scott Robohn (20:28 – 20:29) Right. Carrier product sales team.

 

James Jun (20:29 – 21:48) Carrier product sales team. So a lot of times as a small provider, you don’t qualify for peering, right? So your retail, for all intents and purposes, your retail enterprise. But you also want to negotiate the best outcome for our own customers. So we’re not going to just go and buy a retail DIA from someone like Comcast. You know, not only is the cost horrible, you know, high cost means we’re not negotiating the best value for our customers. You know, we want to provide affordable rates and great connectivity for our users, right? So having right relationships on the sales side, the product management team, for them to understand our business better. So that we’re paying but giving us access to the right folks within the organization. So even though we may be paying you, we’re treated almost like a settlement-free peer from an operation perspective. That is huge, right? So in terms of escalations, troubleshooting, even basic things like local prep. Like we have some of the relationships where we’re a paid peer, right? But we’re configured as a settlement-free peer, right? So those kind of relationships just matter in that having the people communicate those needs and be able to advocate for you. So those matters. And it really comes down to just, you know, get together, have fun, you know, get some beers, break bread, right? Just, you know, that was really… And as a shy and introverted guy, it takes a long time to figure out how to even have those managers kind of people relationships. 

 

Scott Robohn (21:48 – 21:50) Shy and introverted, I don’t buy that for a second. 

 

James Jun (21:50 – 21:54) Well, maybe not anymore.

 

Scott Robohn (21:55 – 21:55) Maybe not anymore.

 

James Jun (21:55 – 21:58) Well, back in those days, it was difficult to transition out of that, right?

 

Scott Robohn (21:59 – 21:59) Sure, sure.

 

James Jun (21:59 – 23:07) So that was an interesting lesson. And as it continues, so one of the biggest thing I think was 2013, around that time is that we were also able to get a local peering with the Internet2 GigaPOP, that’s the Northern Crossroads. So they were run by Harvard, MIT, now it’s run by MIT today. They’re really good people. Brian Murlow, Chris O’Brien, the whole team, they keep a really great shop. And when we got the peering, that was fantastic, because all of our university-bound traffic on our network, it was hit or miss.

We always had to go through. You know, sometimes you have to really pick and choose who their transit providers are. Sure. So, you know, most of the time we try to keep it in Boston. Sometimes it would go through New York. Having that direct peering eliminated all of those problems. And it was really fantastic to get. So, you know, so by the time, you know, of early 2010s was really what we’re really focusing on improving, having the best cleanest looking trace routes, best, you know, adjacencies with other ASNs. That was really, it was that time.

 

Scott Robohn (23:09 – 23:26) So to follow on that, right, so that you obviously needed all the, you had all the activity that got you into providing good peering and access for your customers. Was that the chicken and the egg was getting into the fiber and physical interconnect?

 

James Jun (23:26 – 23:55) Yeah, yeah. That’s a great, great, yeah, that’s a very great question. So it’s interesting, back in 2010s, we were, you know, we’re just network engineers. And our customers love that. We have some of our customers today who’s been with us since we started over 10 years. Sure. You know, they’ve seen it all, right? And they’re very loyal customers. We thank them. And, you know, they love what we do. They love us back then. They still love us now. And, but back then, we had no idea about, I mean, obviously, yeah, fiber, you got the fiber jumper. We know basic things. And you don’t have to use light meter.

 

Scott Robohn (23:55 – 23:59) Oh, there’s jumpers between two racks.

 

James Jun (23:59 – 24:03) But the most experience that we had, the most experience that we had was fiber was looking at a patch panel facing the cross connect, right? 

 

Scott Robohn (24:03 – 24:05)There’s a little more to it than that. 

 

James Jun (24:05 – 24:22) There’s a little more to it. So you got, you know, the most interaction you have is light meter, you know, so it’s like, and then if you’re, and then we started, you know, when we started, as we expanded our network footprint to other data centers, we started leasing dark fiber from, you know, carriers, from our carrier partners, right? So, so that’s when we, you know, and OTDRs kind of price you.

 

Scott Robohn (24:22 – 24:23) If they would lease it to you.

 

James Jun (24:23 – 24:46) Oh, so that was, that’s a good, that’s a great point that you mentioned. Initially, we had a very difficult time trying to get leased dark fiber. It was, oh, I say you’re a competitor. Like there was this one carrier that lost, they told me that they lost $350 DIA deal to us. And I don’t even know, because like, we don’t really, we’re not really a sales-ish kind of company. People usually call us, so I don’t, I don’t recall competing or none of us did. We’re a little confused, but you know, I understand their perspective.

 

Scott Robohn (24:46 – 24:47) It’s an inbound call. You’re going to take it.

 

James Jun (24:47 – 26:16) Yeah, I mean, if it came from the end user, the customer’s not going to say, oh, I’m calling you because I hate this guy. Like we don’t need to, that’s none of our business. Our business is just to make the customer happy, right? So, but you know, I can understand where they’re coming from. I guess that one customer was moving away from them. We had no idea about it. And even if we did, you know, if customers are looking for service, it’s none of our business. We’ve got to provide it, right? So they got a little hurt by that. And when we called them for dark fiber, they said, we’re banned from buying any dark fiber. I was like, oh, well, that’s interesting. So I thought, so we, at the time, so we had some of the struggles back then of, because as a provider that’s expanding to these different data centers across Boston in the I-120 and the, you know, I-95, I-120 corridor, we needed dark fiber. Yeah, we’re a power user of dark fiber. So we, you know, the good news is there were some different competitors that are showing up. So we’re always able to find competitors’ dark fiber where someone was always willing to sell us fiber, right? But it did also make things a little bit difficult when you are banned by the major fiber provider from buying any dark fiber. So we had to do some workarounds. But eventually, we all made up.

We have a very good relationship with a successor of that provider now. You know, I’m glad that people came around. So that’s good. But yeah, so we were lucky that despite the challenges, we were able to find access to dark fiber. But you are 100% right in that what we call getting assured access to fiber assets has always been a moving target in Boston, especially as a small service provider.

 

Scott Robohn (26:16 – 26:17) Got to be frustrating. Got to be frustrating.

 

James Jun (26:17 – 26:50) It could be frustrating. But, you know, it’s one of those things that you just got to navigate. And you don’t always get the best price either, right? And enterprise retail customers get cheaper fiber than, you know, that we could get just because we’re, you know, we’re marked down as a competitor. And in all fairness, I understand where they’re coming from. I’m not complaining. I understand why they would view that way. If I was them, I don’t know if I would behave the same. You know, it’s, you know, I don’t see the point of fighting with competitors. It’s called the internet and telecom. So I don’t see the point of that. But I can at least understand why they thought the way they did, right?

 

Scott Robohn (26:50 – 27:13) So on the infrastructure side now, like one of the things that I caught my eye with you is the stuff you put on LinkedIn, and you’ve got pictures down in manhole covers and conduit. And sometimes the weather conditions are not that great. I mean, we are in the Boston area. Winters are hard here. What does that side of the business look like for you now? Like, you’re glad you have great relationships and you can get dark fiber when you need it.

 

James Jun (27:13 – 28:10) So let’s get to that slowly. And I apologize for being sidetracked here. So at the time, we were very, you know, we had no idea about fiber. The most experience we had was Andrew just plugging in and plug our router, plug our DWDM to, you know, the cross-tech panel, right? And then as the time went on, you know, as we became, by mid-2010s, right, we became probably arguably the most popular bandwidth provider in all of data centers in, you know, Eastern Massachusetts, right? Everyone’s calling, so we had, by that time, we were connected to all the different data centers.

And as it happens, one of our data center partners, who’s been a fantastic partner, I believe, you know, over time, they wanted to also get into become their own IP provider. Okay, sure. And their move, unfortunately, was to say, we’re a competitor, we’re a customer, right?

 

Scott Robohn (28:10 – 28:10) They kick you out.

 

James Jun (28:10 – 29:08) So they kicked us out, right? So that was a very eye-opening situation. And at the time, so this was, I think it was 2018 when we got kicked out.

And that was a very distressing time for us. That was probably, you know, in the years we’ve been in business, that was probably the moment where we’re at our closest to actually going out of business. Imagine, like, if you are, say, in Equinix or some large, you know, ecosystem, right? You have, you’re a power user of CrossConnect, or you have all your PNI’s, your paid peering, settlement free peering, IX, transit, all that. You have riddles and settle, and you have all the CrossConnect, hundreds of CrossConnect coming in from your customers, right? You’re being successful. And all of a sudden, you have to pull out, you have to get out. And there’s no legal recourse. So that was a thing, that was a moment that really opened our eyes up. We learned a tremendous lesson that here in this United States, we, and I love, you know, for its good side and downside, I love this country, right? Don’t get me wrong.

 

Scott Robohn (29:08 – 29:12) And you have perspective, right? You’re not from the U.S. originally, right?

 

James Jun (29:12 – 29:54) Right. But you got a different perspective. At some point, you’re like, we’re in a free country, we’re a nation of loss. How is this legal? How does one business really decide that I’m going to destroy you? Instead of making an offer to buy, I’m just, you know, I just want to put you out of business. I’m just going to steal all your customers.

How is this even legal? How is that possible? And then, you know, so we had a lot of, you know, interactions with our legal folks, lawyers and all that. And, you know, there’s only so much lawyers can do, but they do give you some good guidance in terms of how they look at it from their practice, right? And one of the things that we really learned in the United States, and I think it’s like that in a lot of places, is, you know, when we call someone a landlord, the rule matters. 

 

Scott Robohn (29:54 – 29:55) Okay. Wow.

 

James Jun (29:55 – 29:58) So in that land, you are God. Okay. 

 

Scott Robohn (29:58 – 29:59) That will make the show notes. Definitely.

 

James Jun (29:59 – 30:17) If you’re in my house, and I want you out, sure, there are consumer regulations. If you are a residential, you know, tenant, right? The state has consumer protection and all that. Commercial businesses do not get a lot of protection. So if I say you’re out, I’m not reading your rent. I’m the landlord, my house, my rules, get out.

 

Scott Robohn (30:17 – 30:17) Yeah. You’re done.

 

James Jun (30:18 – 30:54) And, but what’s fascinating too, so wait a minute. So here we are in the data center and internet infrastructure, right? You know, you know, a lot of data centers today market themselves as, oh, we are the digital infrastructure of the world.

You know, you know, we’re the most critical piece of infrastructure that connects the entire world. You know, we’re very important as utilities and so forth. Great for them. They’re not lying. They’re 100%. I 100% agree with them. So they’re marketing how great they are and how important they are to society. And yet they could just make, on a whim, decide to just pick winners and losers. Sure. Because last time this happened was in 1800s with railroads and the utilities. And that’s how these industries all got regulated.

 

Scott Robohn (30:55 – 30:55) Right.

 

James Jun (30:55 – 30:58) So, so. So you’re Wild West, even on the East Coast.

 

Scott Robohn (30:58 – 30:59) Right.

 

James Jun (30:59 – 31:19) So in the data center space, the data center, the cross connection, the interconnections right now is Wild West. And, and where there is really no regulation about that. So if a data center decides to compete with a customer and fight with a customer, that customer is in a very, very big, big trouble. Sure, sure. And that’s, so that really opened our eyes up. So, so.

 

Scott Robohn (31:20 – 31:21) But you came back. You came back swinging.

 

James Jun (31:21 – 31:28) We came back. We came back. Thank God we did. And what’s that saying? What doesn’t kill you makes you stronger. Right. So, so now you’re about to find out. 

 

Scott Robohn (31:29) Okay. I’m ready. 

 

James Jun (31:30 – 31:55 So, so now that, you know, fast forward, you know, we met some great people. This is where going back, relationships are everything, especially in the times of hard time. You know, everybody wants to make money, but people also want to see the underdogs succeed. You know, that’s one of the things that really makes the entrepreneurial spirit of this country really great. And especially in Massachusetts, in Boston, one of the culture is that, you know, we love underdogs.

 

Scott Robohn (31:55 – 31:55) Right.

 

James Jun (31:56 – 31:59) It’s, you know, it’s been deep rooted in Boston culture. Yeah.

 

Scott Robohn (31:59 – 32:04) The Yankees aren’t necessarily appreciated here. And I mean, the baseball team.

 

James Jun (32:04 – 32:55) That’s correct. So there’s that. But you do have a point. So what, so what happened was, there’s a moment where we, you know, there was this, you know, we started getting ourselves more into dark fiber where we started really, what ended up happening was, okay, so if we can’t cross connect with customers and our partners and our providers above ground, then we’re going to have to do it outside of the building. Sure. And so this is the first time where we really sat down is, we really need to figure out how to stop being a loser that just gets banged around by everyone. And we really need to start moving our critical backbone elements outside and be an actual telecom provider. Not because the existing telecom providers are bad. The reason why we were, we never- 

 

Scott Robohn (32:55 – 32:56) You needed the status. You needed the status.

 

James Jun (32:56 – 33:31) The reason why we were never a dark fiber provider because all of our carrier partners rent fantastic fiber networks, right? Even today. These are, you know, these are some of the best people you can find in Boston. And we’re a very satisfied customer. So we never felt the need, like, we stayed on our lane. We were a great IP network, you know, we ran the IX and all that, but we never felt the reason for us to be in the fiber business. But now for strategic survival, we found ourselves, we have to go outside, not because we want to compete with our dark fiber provider partners, but because we need to move our integration outside because we can’t trust the data centers anymore. Okay. After what they’ve done to us, right? So.

 

Scott Robohn (33:31 – 33:35) So what, how long did that take? What was that process like?

 

James Jun (33:35 – 34:00) Right. So this was a very interesting process. So in, so this process started in 2019. And then, well, actually 2018 is when it started. So in 2019, we had our first fiber ring going for our own backbone. And then we were able to completely survive.

And then we, that was 2019, end of 2019, beginning of 2020 is when all of this drama finally settled. We’re like, oh, the storm’s not over. You know, now we have sunny days outside again.

 

Scott Robohn (34:01 – 34:02) And you had another storm. Yeah. 

 

James Jun (34:03 – 34:55) Well, well, no more storms now. Thankfully, thank God. Not going to, so far, so like beginning of 2020 as the COVID started, like terrible time for that. But that’s the moment where everything stabilized. You know, now we have the ability to pretty much get to anywhere else, our fiber networks up and running. And that really also, you know, as nerds, every time these kinds of things happen, it being an IP network, just a lit, you know, networker, right? Sure. And putting yourself into a position where now you’re outside in crew, contractor vans, fusion splicing with them, and seeing all this stuff happening that you’ve never faced before outside. There’s a lot of things that process that go through your brain, right? It’s really fascinating how, it’s just like, you know, you had, you did a great intro where, you know, where a lot of us in the networking space, we’re very fascinated about how, because we take things for granted.

 

Scott Robohn (34:56 – 34:56) For sure.

 

James Jun (34:56 – 35:33) You plug in a cross-connect, it just comes up. You plug in a fiber, it just comes up. Dark fiber, oh, just tubing the ground, it just works. Right. We, but seeing how the sausage is made. That’s a lot that goes into the sausage making. Oh, it’s, yeah. So it was really fascinating. So in 2020, we said, okay, now that we survived, we can also, this can happen again. It’s like, the thing is that with data centers and real estate, there’s a saying that the internet runs on real estate. And the reason why is you need space to put equipment. I mean, you can’t really have routers running in the middle of the street, right?

 

Scott Robohn (35:33 – 35:35) The internet exists in time and space, right? 

 

James Jun (35:35 – 36:26) Yes, the internet exists in time and space and the space are the landlords, right? So we run on real estate and that part is wild west. It’s completely unregulated. This is going to happen again. It’s like every time someone’s rent is, or one of our carriers rent is, oh, we’re going to get kicked out again, again, again, and again. Like this can happen. So it came down to, what it came down to essentially is we love IP networking so much. It’s our dream. We just love the internet. No one’s going to take this away from us. And when someone tried to take that away from us and said, go out of business, you will never run a network again, or you’re just going to become a, just go work for, I don’t know, Google or whatever. We’re not going to accept that, right? So we’re not going to let someone take our dream away from us. So that is how we really dove head in and ended up finding ourselves. Fast forward to today, we’ve ended up finding ourselves becoming a utility company, not just fiber provider, right? So as we- 

 

Scott Robohn (36:26 – 26:27) Wait, do you mean that literally? 

 

James Jun (35:27 – 36:31) We mean that literally, yeah.

 

Scott Robohn (36:31 – 36:34) So how are you a utility company? Are you regulated like a utility?

 

James Jun (36:34 – 40:29) We are, yeah. So what happened is after we survived by running our fiber network, we said, we need a bigger insurance, just us surviving. Next thing what’s going to happen is these landlords are going to start kicking out our partner carriers, or they’re going to start doing something else, right? As long as our partners, our biggest providers and vendors are relying on landlords, they too are at a whim of getting kicked out. This could happen. This is going to be a repeating circle of insanity, right? It’s like we have to break the cycle, right? So how do we do that? And then we finally figured it out.

One of the things that really fascinated was… So I’ve always been neutral about cross-connection. A lot of us in networking space, we always complain about cross-connect cost, and I understand why. As a small business owner, right? Cross-connect costs add up real quick. I mean, Disney’s cross-connect costs more than the dark fiber nowadays, right?

It’s pretty expensive. But at the same time, when you truly appreciate how the internet works, all the stuff that goes in there, yeah, cross-connect is a simple thing. It’s just a pair of fiber that is very profitable for the data center. But what it also represents is the ecosystem, right? The relationship building, the customers, the investment they’ve made in the entire platform of their building. So there is a value in cross-connect. So I’ve always kind of viewed that neutrally over time. But to be fair to the people that do complain about it, though, as we livened our fiber ring, now we have become a full-blown dark fiber company. One thing that really fascinated me differently was, oh my God, we’re saving so much money on cross-connection. This is great. Because we’re doing our cross-connection out in the manholes. I’m like, wow, this is fantastic. So but then it also, a light bulb went in our head. So thus far, all the, like us getting kicked out and our carriers being threatened and things like that. This happens because us in the telecom industry have made itself vulnerable by relying on real estate. Because we’re always seeing each other as competitors, right? So we couldn’t work with each other, right? So what happened was we got ourselves into real estate owners because we felt that they were neutral. And they were. Because the real estate owners didn’t sell internet. They don’t do telecom, right? They’re just landlords. So they don’t have a skin in our business. So they were naturally the natural neutral party for us to build our networks to and cross-connect and interconnect, do all of that. But now we live in a space where a lot of these landlords are now becoming their own network providers. So eventually, at first, it started by having discriminated cross-connect rates. Like if you buy the landlord’s own internet or telecom service, you don’t pay cross-connect.

That’s like $500 a month. But if you buy one of the carriers, then, you know, they can do whatever they want, right? There’s nothing wrong with that. But then for us and for our partner carriers, when we are being threatened and when we’re being kicked out, that’s a different, that’s disruptive, right? So what really occurred to us was, okay, so we’ve become vulnerable now. It’s not even just a space and power for it. Network equipment does not generate that much power, right? If you, worst case scenario, if you really wanted to jerry  rig it, you know, just find space, do your own DC plant, right? You can make that work.

Carriers do that all the time. You know, that’s how, it’s a central office mentality, right? Sure, sure. So we’re not like enterprises or cloud or AI that draws a lot of power. So while we value the protected power environment of data centers, we are, because we’re such a small network provider, such small footprint users, we can do it ourselves if it had to. So, but what really made us get vulnerable is the cross connection. We have to interconnect with our customers, with our vendors. So when we started doing it around the manholes, it all of a sudden, what used to be our most vulnerable Achilles heel, just because we’re fixed to that building, now we became mobile.

 

Scott Robohn (40:30 – 40:32) Totally removed that from the equation.

 

James Jun (40:32 – 41:27) We’ve taken the most important mode, the most important power advantage that landlords have, we took that away from them. Now, to me, to us, I would love to work it, like the data center that we’re in, a fantastic part of it. They run top-notch operations, great team. We have great relationship with them. That’s why we’re here, because where was it? But we don’t necessarily have to be data centers anymore.

Our network head end could be any commercial office. We could build our own DC plan if we had to. Obviously, we would prefer to work with data center. We like to stay in our lane, right? But the point is, even with data centers, now data centers know too, just in this place that we’re sitting at, the building next door is another data center, right? So now you have this competitive ecosystem where the core of where we make the interconnections is no longer fixed to an address, right? So now it’s mobile. So if someone threatened us, oh, just pull the cable, we’ll move it to that building, and whatever.

 

Scott Robohn (41:27 – 41:28) Moves it. The connection’s over there.

 

James Jun (41:28 – 42:08) The cross-connection’s outside. So what we’ve said is, throughout 2020, as we’re going through that, we’ve said, okay, we’ve seen all of this. Now to really make the difference is taking our experience and our story, now let’s democratize it, let’s product it, let’s package it, let’s weaponize it, and let’s throw it to the wild. Because I will tell you right now and everyone, I’ll make it my last mission that no landlord of anyone is ever going to screw an internet business ever again.

 

Scott Robohn (42:08 – 42:10) You don’t ever want to be beholden to that kind of relationship again.

 

James Jun (42:11 – 43:03) Not just for myself, I don’t want to see anyone else get hurt by a realtor ever again for the rest of our career. And at the very least, in the markets that I operate, not happening. So Boston, I love Boston. I grew up here. It’s my home. Sure.

Right? Yep. I don’t want to see that ever happen in Boston ever again. So that’s where, so far, the most construction we ever did was we had a great consultant, great engineers working for us, and they have helped us a lot. So the most biggest conduit system we have was just for our fiber network, right? Four-inch conduit, and then you had to do what’s called a joint trench. That’s the cities, there’s a regulation within the city to prevent the streets from getting too crowded. So you go through the process, you get your conduit chain, you have some small manholes to put your spice case in here and there. So it was a very typical fiber belt, right? And it was very fascinating. We love every minute of it.

 

Scott Robohn (43:04 – 43:12) And it’s even more complicated than that, right? Because you have the municipality of Boston, but you also deal with Somerville and Cambridge.

 

James Jun (43:13 – 44:48) Yep. Everyone has their own different rules, right? So from that point, so when it was just for our fiber network, we could easily work in the existing municipal regulations and their assumptions of how utilities construction is typically done, right? But now, we’re trying to democratize, weaponize, and release it to the wild. Then suddenly, this small three-by-three handholes and one or two conduits is not going to be enough because now we find ourselves in a situation where it was really strange talking with all our fiber providers and our partners. Everyone’s like, why would you want to do that? Why would you want to help your competitors? And I tell them, listen, we’re not trying to build a fiber network here. We found ourselves in a situation where we actually need our competitors, our friends, our enemies, even people that absolutely hate our guts to actually come into our manholes so they can cross-connect with me and everybody else.

So now, we’re no longer in the fiber business. Now, we’re in the rail space and it’s a tunnel, right-of-way business. Now, we are in this utility space. So OK, so now, this is completely something else, right? So we had to, first and foremost, we had to find a place that would host this idea, right? This is going to be a massive disruption we are causing to public streets. It’s huge. It’s a big dig, right? So right from the get-go, City of Boston was initially not a good candidate. Nothing against the City of Boston. It’s a fantastic partner. They love what we’re doing. We love them. But the challenge of the city, as you would imagine, there’s a lot of old infrastructure.

 

Scott Robohn (44:48 – 44:49) Old infrastructure.

 

James Jun (44:49 – 44:50) Old infrastructure. There’s a lot of congestion.

 

Scott Robohn (44:51 – 44:56) You know, if you know 93, it was a cow path and now it’s an interstate. 

 

James Jun (44:56 – 45:59) Exactly. And a lot of subways, like the Red Line, the Green Line, they have such little, even the water may be so shallow because when you start digging up those streets, you’re not digging up soil. You’re digging up the ceiling of the subway. You have literally no space. So what we needed was, we needed concepts, kind of a mini version of Manhattan versus New Jersey, if you look at the New York market. Manhattan has these great carry hotels, fantastic assets, right? But then you also have, if you really want heavy infrastructure in the data center, everybody goes to New Jersey because of it being slightly suburban. It’s not suburban, it’s New York. After 9-11, they moved a lot on the other side of the river, yeah. But part of that is also because it’s an industrial district, building new infrastructure is generally a lot easier. So in Boston, looking at the concept, Cambridge was a potentially good location. But then we also saw, we liked Somerville, this location where there’s a place called Innerbelt. And what was unique about Innerbelt was Innerbelt really shined in the dot-com era.

 

Scott Robohn (46:00 – 46:06) And just like your Ashburn, is that a fair analogy? 

 

James Jun (46:03 – 46:06)On a very, very small, maybe one thousandth.

 

Scott Robohn (46:07 – 46:08) Sorry, I’m from Virginia, right? So Ashburn. 

 

James Jun (46:08 – 46:48) But you are correct, yeah. So it’s kind of like, yeah, so I generally tell people it’s like a mini version of New Jersey versus Manhattan, right? So you’re right next to downtown. So for all intents and purposes, you are downtown, but it’s industrial, right? It’s easier. So in 2000, when 360 Networks back then, which became Hibernia, when they built the subsea cable from Lynn, Massachusetts to Halifax and then out to Ireland, 360 also built a fantastic terrestrial route that connected their cable landing station. And if you look at the shoreline of Boston, Lynn makes a lot of sense. You can’t bring a cable to Boston.

 

Scott Robohn (46:48 – 46:48) Sure.

 

James Jun (46:49 – 47:56) There are some things that get in the way. So you got to bring it to Lynn, right? So they built this terrestrial front hall route from the Lynn landing station to the downtown ecosystem through the MBTA, the commuter rail right of way. So you can’t get any more direct than a commuter rail. You know, it’s very direct, right? So it’s like 15 kilometers from there. And it comes into Interbell, where we’re sitting in today, because we’re right next to railroad tracks. So and then as we’re looking at this, the data’s there. This location had two data centers, which also is important for us because we don’t want to just do work for one data center, making a significant investment in upgrading utilities, because that one data center, if we ever had a falling out with them, it’s a naturally contentious relationship. And we don’t, you know, we’d rather prefer that we have more than one data center that we can kind of bank on, right? To create a little bit of the ecosystem, the competitive spirit, if you will, for our customers and for the good of the market. So Somerville started looking very attractive for us. It’s industrial area. There are two data centers clumped together. And then there’s a subsea cable, the terrestrial route coming from the landing stage.

 

Scott Robohn (47:56 – 47:58) Those three things lined up, yeah.

 

James Jun (47:58 – 48:09) And you’re surrounded by, being surrounded by railroad is a curse and a blessing at the same time. It’s a curse because you want to build a new route out of that. Oh, you’re going to get railroaded. It’s not easy to cross railroad. So there’s a curse.

 

Scott Robohn (48:09 – 48:09) Sure.

 

James Jun (48:10 – 48:15) But it’s a blessing that the long hauls are, you know, it’s so much easier to get the long hauls because they’re right next door.

 

Scott Robohn (48:15 – 48:18) Like Sprint’s famous for like, all their long haul was around CSX.

 

James Jun (48:18 – 50:48) Exactly. So, you know, if you’re in the long haul business, despite all the complexities and the bureaucratic BS that you have with railroads, it beats having to deal with a gazillion different towns along the way. So it’s a blessing and a curse. But for what we’re trying to do, for the global internet traffic, railroad is actually a friendly thing, believe it or not. Right. So ask me about this. So we first started approaching the city of Somerville, where we are. And Somerville has been a fantastic partner. You know, they’ve been, they got it. They got it. As a matter of fact, Somerville’s interesting is that Somerville was the first city in Massachusetts where they had competition in cable, the broadband. I think it was, I think it was like 97 or so. So you had RCN, I think there was another, I think it was Media, one was the other at the time. Or so like this was the first municipality where you could, you had choice of two cable providers back then. So they were, so Somerville was always unique in that, you know, they were always a kind of outlier, you know, they’re not afraid of innovating or taking risks. So, you know, Somerville DPW, when they looked at our initial proposal, obviously their first reaction is like, what the hell are you guys doing? Right. And I don’t blame them because, I mean, we’re proposing a substantial amount of disruption and I don’t blame them. If I were the municipality, I’d be like, I would freak out. Right. But, you know, we continue the conversations over time. Right. And they understood what we’re doing. They saw what we’re doing inside like a major joint trench or the big ones. Right. Where, you know, from a city’s eyes, even though they are not internet or network engineers, they don’t, you know, they don’t see the network like the way you and I do. Right. But they are also, they also understand the basics and they’re also good civil engineers. They know how to manage it right away. Right. So from city’s eyes, one thing that cities absolutely hate is that why is it that, you know, if you’re in TPW, if you’re a civil engineer, naturally, you’re going to love monopolies. And I’ll tell you why. Okay. If you have a problem with water main, the city runs water main. One place to go. One guy to go call the water department. Sure. If the electric duct bank blows, call the power company. Yep.

If there’s a gas problem, call the gas company. But somehow with telecom, you call Verizon and Verizon, oh, that’s not my manhole. And then there’s, and then why is it that with telecom, there’s 15, 20 gazillion different companies all proposing to dig the streets, to tear off the streets. And every time someone puts their pipe in, you keep losing space. The road’s only 50 feet wide. And by the time you’re done, it’s like the road is complete. It’s a total disaster with telecom. Wow. So from city’s perspective, if you ask Boston, Cambridge, Somerville, anywhere, even any random town. 

 

Scott Robohn (50:49 – 50:50) That’s got to be real pain for them. 

 

James Jun (50:51 – 51:34) Oh, it’s real pain. That’s real pain. They’re like, what the hell? And they hate the fact that these telecom guys are all competing and they just can’t work. They can’t play nice. Everyone’s, oh, you’re not allowed in my condo. We don’t sell. So it’s like, so cities are left holding the back because us telecom providers are so dysfunctional. We always fight with each other, that the cities end up having to deal with the fact that the road keeps getting torn up all the time. Right. Because everybody just can’t. So like Boston got fed up with it. Cambridge got fed up with it. Even Somerville got fed up with it. They’re like, OK, everybody that do build, now you got to do what’s called joint trench. So when one telecom provider says, I’m going to put my line in, my pipe in there, now you got to do a public notice, right? You got to get notice out. You got to take out advertisement. You need to notify all of the insurance telecom providers and say.

 

Scott Robohn (51:34 – 51:35) And anybody who wants to put fiber in?

 

James Jun (51:36 – 56:18) You got to invite them to join into a construction. So they too can get a pipe in. But the challenge with this joint trench is that in Boston, joint trench also became a natural racket, which is kind of hilarious. So the way the telecom providers reacted to the joint trench rule to take advantage of is to create this little middle school sophomore club, if you will, like an old school boy club, right? So what happens is these telecom providers, when they do the joint trench, they send an invitation. And the reality is, if you’re a new provider, you get a joint trench letter inviting you to construction. Now you have to take a look at this. Because if you don’t have a customer or if you don’t have a network plan in that particular area at the time, why spend $75,000 or $150,000 to share the construction cost? That’s a lot of capital. So most likely, if you’re a new ISP or even if you’re an existing ISP, unless you have a customer or some kind of project going at the time, a lot of providers opt out of the joint trench because they just can’t afford the capex right now. So some providers jump in, but a lot of providers don’t jump in. So then after construction is done, the streets are repaved, when that other provider now needs fiber, the existing providers that were part of the joint trench are not going to let them in, right?

So now we’re back to square zero again. Now the new providers are being locked out of the streets again. So then what happened was these existing members of the joint trench run this little racket clubs where, oh, you scratched my back. I know you. You know me. Buy me a beer. Oh, here, here. There’s a running joke. It’s called a 40% to 80% legit conduit licenses, right? Where there is no license. There’s no title. It’s like, oh, I know it. Yeah, I hear it. Yeah, just pull the cable. Yeah, yeah, tell this guy that you know me. That’s kind of how a lot of people started running fibers around here. But if you’re a new kid on the block and you’re not one of them, no, you’re never getting in, right? So that’s kind of what’s been going on.

When we started this project to bring the cross-current ecosystem out to the streets, we couldn’t work with the existing joint trench model. And Summerville was a great partner for this. Because what we wanted to propose is not only do we want to do a joint trench while the cables jump in, we’re also going to put in future capacity. We’re going to own those capacity. We’re going to be a regulated utility under the Massachusetts Department of Telecom and Cable, where we’re going to have the published, under the tariff doctrine, the published attachment rate. So we’re subject to the same pull and conduit attachment regulations as the power company and Verizon, right? So the value is, once the system is put in, anyone who did not participate in the joint trench can easily put their cable in because all of the conduits that we own and operate are openly leased out to everyone for the duration, for the lifecycle of the entire system. And the city loves that, right? Sure. You know, they’re like, oh my, this is a joint trench that is finally well run without the dysfunction, right? Sure. So from a regulatory perspective, I think that we were very lucky and blessed to have found a fantastic municipal partner, right? Where they were really smart. They understand the challenges of managing the right of way from a telecom perspective. And they really welcomed this kind of idea, right? So it was a great partnership to work with them. And over time, as we started this construction, it was a lot of capital, right? Very expensive. Sure. And we’ll get to some of that. But the value is, fast forward to today, now that the system’s up and running, because of that open access system, right? We have a situation where not only the existing telecom, not only the old school boy club guys are freely running their fiber in our system. You also have non-traditional fiber. You would have international networks that are becoming, like everyone and their grandmothers becoming a dark fiber provider now in Boston. It’s hilarious. It’s almost like Boston has gone from one of the most awkward cities for internet infrastructure, and it’s now turning into the network engineer’s almost like wet dream, if you will. Where the becoming a dark fiber provider has become so accessible. Interesting. Because Massachusetts, believe it or not, Massachusetts from a regulatory perspective, becoming a SELAC, becoming a SELAC license to have the ability, authority, if you will, to own and run your own fiber cable in the ground. Massachusetts is one of the most telecom friendly states in terms of regulations.

 

Scott Robohn (56:19 – 56:31) You made that comment to me back at ITW in May 2025. And it’s like, I wasn’t sure about that. But you’ve clearly explained it. Are there any other metros in the US that are coming close to?

 

James Jun (56:31 – 56:44) I think a lot of them are. Like North Carolina is another state that’s fairly open. That’s another one out there. Florida is interesting. They’re also pretty fairly easy to work with. Some other states are a little difficult. Like Texas has some challenges.

 

Scott Robohn (56:44 – 56:46) That’s interesting and surprising, actually.

 

James Jun (56:46 – 57:21) That is surprising. You would think that. But yeah, in Texas, there’s a bit of a right of way compensation that you have to do, from what I understand.

I’m not 100%. I mean, I’m not a lawyer. But it’s not surprising. We found it surprising that Massachusetts is easier. New York’s challenging. New York depends on location. New York is New York. New Jersey is not too bad from a regulated perspective. But what happens is the regulation makes a lot of sense from a regulated perspective. And that’s good for them. But a lot of the challenge that we see in broadband infrastructure is not so much the regulations per se. It does matter. I’m not ignoring that. It does matter.

 

Scott Robohn (57:21 – 57:23) Clearly, from everything you’ve said today, it does matter.

 

James Jun (57:24 – 58:35) It does matter. But what I mean by that is before the regulator even becomes your number one problem to deal with, the number one problem is making a business essential. Lack of capital versus making a return. Sure. Right? So becoming a SELAC is so easy in Massachusetts. Well, then why do we have such dark fiber? Why was it so difficult to find dark fiber? Because there’s no competition out there, right? And the reason for that is in Boston, sure, becoming a SELAC is easy. Yeah, you got your SELAC license. But now you need conduit. You need access to conduit. And everyone has this old school racket club going on that’s blocking out. So then, yeah, you’re a SELAC. Congratulations. Guess what? You’re not coming to my conduit. Yeah, you’re still on the back side. Right, you’re still blocked out. That’s not the fault of the state or the regulators from a state law perspective. But there are some deficiencies in terms of how the conduits are regulated on the municipal levels maybe. But I think before we even put that on the municipalities, I think it’s really the first fault lies in the telecom providers that are naturally against competition, right? And when it’s natural, and it’s not like they were deliberately doing this either. These are all good-minded people. Like they’re not trying to deliberately create- 

 

Scott Robohn (58:36 – 58:38) The way an ecosystem emerged. 

 

James Jun (58:39 – 59:14) Yeah, it’s ecosystem emerged. It was a natural inefficiency. Some people started using it as a loophole. Some people didn’t. But as it happened to the way it was, then as all of us started seeing this problem, it already became, oh, well, yeah, it’s not great. But you know what? Maybe it’s a feature, not a bug. Let’s just leave it alone. So that’s kind of how it transpired. And what we’ve built is, because it’s an open access system, it’s really shattering that whole model, right? And so it’s interesting how we started from being networkers and fast forward to today and now we’re a utility.

 

Scott Robohn (59:14 – 1:00:14) So let’s use that. And we do need to start to wind this down. But again, from my early interaction with you and thinking about, well, all the challenges are actually in building out the physical infrastructure, you’ve added a dimension. Like over the time I’ve gotten to know you, the business and the regulatory pieces are at least as important and maybe more complex than actually what it means to build a conduit, trench for it, and put fiber through it. Those problems seem pretty easily solved compared to having to go out and go from being, what did you say? Introverted and… introverted or opening yourself up. And you’ve laid out a path of, you had to learn to work with the carriers and sell your vision and what you’re trying to do with them. You needed to work with regulators and city officials to say, we need to change the system. And for those capital intensive projects, you probably also needed to work with investors or however else you raise capital to do this stuff.

 

James Jun (1:00:14 – 1:00:15) Right. 

 

Scott Robohn (1:00:15 – 1:00:37) So what did I miss? What else is big there? What would you tell people who are where you started? Okay, I understand how peering works or I support access networks. If people want to get into what you’re doing, what are two or three things, what wisdom could you drop on them to say, these are things you need to start learning about?

 

James Jun (1:00:37 – 1:02:53) Yeah. One thing, yeah. To be honest with you, I’m not sure if I’m the right guy to be giving wisdom stories, but I will tell you this much. I’ll tell you my experience. Your perspective, your experience. It’s all you can speak to. That’s right. All I can speak to is we are very tenacious and we’re very, I’m not, I don’t think stubborn is the right word because we’re very flexible, but we’re also very, we have our, you know, we started this company because we love TCP/IP, not just networking. Sure. Just IP is, when you look at IP and I think that, you know, when you talk to Vint Cerf, he will probably tell you a little bit similar theme, I think. I would love to chat with him someday, but when you look at the history of IP, IP represents freedom. We are breaking away from the old circuit based, you know, telecom mentality. It’s freedom to whatever you want, right? That’s truly, and if you really look at the societal and historical nature of what internet really is and what telecom and communications really is, you start understanding, you start to realize why USPS was owned by the government from the days of George Washington. Communications, information wants to be free. So not to get too sidetracked, I’ve been a very huge fan of IP and it’s my dream. It’s my baby. I’m not a large company. We run a great network. We have fantastic customers. They love us, but it represents what I love getting up to work every day and doing. And when someone told me, I’m going to take it away from you. That became, that was so, that was so traumatic. And, and, and I said, no one’s going to take that away from us.  So, and very tenacious.  As you can see, you can see the story has spent many, many years, right? So we never gave up. We were very persistent in our vision, in our, in keeping our dream. And that, and we, we would always get beat down as a small company, but we always got up. Right. And we never gave up. 

 

Scott Robohn (1:02:53 – 1:02:54) Definition of tenacious right there.

 

James Jun (1:02:55 – 1:03:12) We’re very tenacious. We’re like that angry people that’s biting something, just never letting go. We just never let go. So now that to your, to what you were mentioning earlier, what happened in the past, in the span of past 10 years, we’ve come out into a completely different person.

 

Scott Robohn (1:03:12 – 1:03:16) Yeah. You’ve had to learn so much about so many different facets of this.

 

James Jun (1:03:16 – 1:03:26) Yes. Out of this, all because, to me, I can tell you right now, the most number one killer app on the internet is TraceRoute. I’m sorry. It just is what it is. Okay.

 

Scott Robohn (1:03:26 – 1:03:33) You might be the only person standing behind that assertion, but anyway.

 

James Jun (1:03:33 – 1:04:45) You’re not wrong. But the point is, okay, listen, you know, we, we love IP networking. Sure. And you know, when, you know, and the reality is we’ve seen, Boston’s a unique and very difficult market in telecom it used to be. I think things are getting a lot better now. It’s a vicious market, as you’ve seen, right? But so really, you know, Boston has really changed us in ways that we never thought we would. It’s going through all this experience and we’ve survived and now we’re a strong player in this space. And, you know, everyone that we’ve been through, even our enemies, even the folks that kicked us out, I can tell you, I appreciate every one of them because look at where we are today. They’ve made us better. So we survived. Now, you know, you know, now you got even subsea cable guys coming into the manholes doing their own thing. You got CDNs having supplies. I mean, can you imagine some hosting shop playing dark viber? I mean, who would have thought, that doesn’t even make any sense, right? It’s crazy how Boston has gone from one of the most, you know, there’s old school boy club, racket club, kind of closed telecom society to it’s just completely open, right?

 

Scott Robohn (1:04:46 – 1:05:09) That’s quite interesting. I’m not sure we have a better place to stop the episode. So like, that’s an awesome wrap. I do have like one fun question that I ask everybody on the podcast. What’s the worst outage you’ve ever caused? And you have some multiple facets of experience here. So you’ve been doing this long enough. You’ve caused an outage, right?

 

James Jun (1:05:10 – 1:05:48) There are two outages that I can remember that will be the worst of my career. The first is back in the old days when I was a kid, right? The same outage that everyone was into. We all get, you know, classic iOS made all of us better. Sure, yeah. Where one typo brings down the whole network. Everyone’s been through that. I had that moment happen to me over an ACL, right? The classic. It’s a story for everyone, but you learn very quickly not to do that ever again. And that’s why a lot of us in network, old school network, like including myself, we’re all Juniper fans. Well, we started with Juniper because when Juniper came in, they introduced the concept of commit. So that was really… commit and rollback. 

 

Scott Robohn (1:05:48 – 1:05:50) Commit and rollback. Saved your bacon so many times. 

 

James Jun (1:05:50 – 1:05:57) It is. Let me tell you, it’s just that one feature. It made the internet a lot more reliable, what they did.

 

Scott Robohn (1:05:57 – 1:06:10) That was like the one selling point. First, I had a friend who went early to Juniper as a resident engineer at UUNet, who said, we have rollback. You need to come take a look at this. Absolutely.

 

James Jun (1:06:10 – 1:06:17) That one feature, I think, is one of the least talked about heroic features that made the internet more reliable.

 

Scott Robohn (1:06:17 – 1:06:17) Now it’s everywhere.

 

James Jun (1:06:17 – 1:06:18) Single-handedly.

 

Scott Robohn (1:06:18 – 1:06:19) It’s just a good idea.

 

James Jun (1:06:19 – 1:06:38) Even Cisco, it’s interesting. You know, I grew my career out of Juniper. But today, what a Cisco shop. We’re all ASR 9K, we’ve got 8,000 coming in, NCS, all that. And iOS XR, we’re a big fan of that. It’s got the Juniper-like features. I don’t get it wrong. Well, I’m always a Juniper fan.  You’ve got options.

 

Scott Robohn (1:06:38 – 1:06:40)You’ve got options. That’s right. 

 

James Jun (1:06:41 – 1:09:37) To your point, it’s everywhere. You’ve got Nokia, even Arista’s got this feature now, which is fantastic. So that’s one thing. And then the other outage was when we started doing a lot of automations, with our own automation tools and all that. So automation is fantastic, right? When you’re provisioning, you’re eliminating a lot of human error outages. So when we do provisioning, the system just goes and does it. No one’s touching CLI. Well, we still touch. We always touch CLI, but we rarely have to do that to turn up a customer, which is great. But oh boy, the scope of the outage just completely went 1,000 times, right? So now you have a situation where you know your scripts and your code, right? If you have that, then that’s great. But you don’t know the stuff that’s going on the vendor side. And you look at iOS XR, or you look at the modern day Junos, you get it. You know, one complaint that I have with these vendors and the networking community, my only complaint, I’m not a smart guy. I’m just saying as an end user, I’ve got to be honest with you. Yeah, you know, iOS Classic and all that back in the old days was really unreliable in many ways. But I’ll tell you, it was simple. And the monolithic stuff, when it ran, it just ran until you hit a bug, okay? Sure. That was that. Today, things have gotten a lot more reliable in many things by the distributed nature of it. Sure. But I got to be honest, now we’re doing VMs and we’re doing all this stupid crap on the routers. And you know what’s happening? There’s like 15,000 different software running on a router. Nobody even knows where the bugs are anymore. It’s more complex, for sure. It’s way, it’s got this way, you’ve got a full-blown ecosystem. It’s like you’re running a cloud provider on your routing engines.

But by the time you’re done, even the vendor doesn’t even know what’s going wrong anymore. So automation running, a simple thing. You know, configure it, bring up an EVPN.

Customer suddenly just bombs off the entire line card. Oh, we never saw that happening. So now, when you have an automation, we had that happen at one point. So now we should like, that really creates, leaves a mark, just like the ACL thing that everyone used to have with the classic iOS, right? So now we’re like, we’ve gotten to a point where now, even with automation, where like, any change, now every time we do network design, one of the design principles that always comes to meetings is more than just about redundancy and physical hardware, all that diversity. It’s now, one of the metrics that we always discuss is what’s called a failure domain. Right. So we also have to diversify not only for physical and hardware standpoint, but also by the use case. Because if there’s an outage, we want to make sure that we limit the scope of that blast, right? So if someone’s, if there’s a L2 VPN or, you know, converged, you know, Metro EPL customer that you’re working on, we probably don’t want outage related to that affecting IP transit customers, right? So now we have, so what ended up happening is we started with, now we’re going back to having all kinds of routers everywhere now, because we wanted to limit the failure domain.

 

Scott Robohn (1:09:38 – 1:09:49) So I have this idea for an episode or set of episodes on design for operations and design for simplicity. Right. So you’ll be back. Maybe we’ll do a round table for that. We’ll get some other operators on. 

 

James Jun (1:09:50) Yeah, I would love that.

 

Scott Robohn (1:09:51 – 1:10:05) I’m signing you up right now. Appreciate it. James, this has been awesome. Before we break, where can people find you? Like, are you out there writing as their other, is there material online where you want to point people to?

 

James Jun (1:10:05 – 1:10:32) Yeah, right now the easiest to find me is on LinkedIn. I am planning on, you know, obviously, you know, I get asked a number of things, you know, to share our experiences and know how just share our stories. You know, right now I’ve been just seldomly from time to time, I’ve been using LinkedIn articles. The articles have not been very great. You don’t get a lot of, you know, viewing exposure versus simple posts. So I might do a separate blog. I haven’t really thought of it yet. But in the interim, people can find me easily on LinkedIn.

 

Scott Robohn (1:10:33 – 1:11:01) That’s how this all started. I saw your posting and got really interested in what you were doing on LinkedIn. So I’ll say plus one to that. Awesome. James, thank you so much. Thank you, everyone, for listening in on yet another conversation on Total Network Operations today. We appreciate your time. We’d love your feedback. If you’ve got something to say about NetOps or you want to hear what’s coming, contact me via DM on LinkedIn or send us a follow up at packetpushers.net slash followup. Thanks again, and we’ll see you next time on Total Network Operations.