It comes down to the basics of the internet - it's very expensive to provide for giving everyone their data at the fastest speed at all times - so they don't - ISPs try and provide for the average usage and let people accept the occasional spikes in demand and slower speeds at those times.
There's
graphs here showing the internet data usage by hour across India - there's a spike at 5pm, and there's lower usage in the early hours of the day.
For gaming, you need your packets to get to the other user as quickly as possible - so when there's a lot of users, you just fall down the queue or go missing entirely - which means the game can't keep up and the experience gets worse.
That graph is pretty much the same everywhere in the world - 5pm is people getting home from work/school and getting online. Then if you add to that playing against someone overseas - you're adding local congestion to the distance problem - at both ends.
I'd also again make the point - wireless connections add a lot of latency. If at all possible, run an ethernet cable to your console/PC when playing online games - those split seconds make a difference.
One side could be better or worse than the other - I'd suggest just looking at what parts of the game are slow - if you get the ball taking ages after you bowl it to actually see it bowl, that's the batsman's connection; if you hit it and don't see anything in the field until it's too late, that's the bowler - trying to sync their side up with where you hit it.
Against all of that, the game needs to be fast enough so that you can see the other player set off for a run and know which end to throw at, and then have both sides synced up as to whether you make it or not.
The extra little bit of traffic at peak times is enough to push that over to being unplayably long - or at least, cause weird visual mismatches - for me the worst is watching a ball clearly go past a fielder and then getting caught or run out - with the replay showing it warping back to the fielder. Luckilly that's usually balanced by the throw from the fielder and the kludgy movement of the wicket keeper being enough to complete a run.
People using automatic fielding online might have an easier job of it in terms of getting the right throw off - but the AI fielding is idiotic at picking the right end to go for, so it balances out.