Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3798 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.8275. Min Range: -0.8275

Max Act: 0.8275. Min Act: -0.0000

Data Index: 624850 (C4 (Web Text))

Max Activating Token Index: 859

Text #1

Max Range: 0.8275. Min Range: -0.8275

Max Act: 0.7547. Min Act: -0.0000

Data Index: 1009474 (C4 (Web Text))

Max Activating Token Index: 527

Text #2

Max Range: 0.8275. Min Range: -0.8275

Max Act: 0.7621. Min Act: -0.0000

Data Index: 1033470 (C4 (Web Text))

Max Activating Token Index: 121

Text #3

Max Range: 0.8275. Min Range: -0.8275

Max Act: 0.7159. Min Act: -0.0000

Data Index: 216361 (C4 (Web Text))

Max Activating Token Index: 579

Text #4

Max Range: 0.8275. Min Range: -0.8275

Max Act: 0.7179. Min Act: -0.0000

Data Index: 207239 (C4 (Web Text))

Max Activating Token Index: 839

Text #5

Max Range: 0.8275. Min Range: -0.8275

Max Act: 0.7537. Min Act: -0.0000

Data Index: 846264 (C4 (Web Text))

Max Activating Token Index: 681

Text #6

Max Range: 0.8275. Min Range: -0.8275

Max Act: 0.6682. Min Act: -0.0000

Data Index: 239083 (C4 (Web Text))

Max Activating Token Index: 528

Text #7

Max Range: 0.8275. Min Range: -0.8275

Max Act: 0.7584. Min Act: -0.0000

Data Index: 917172 (C4 (Web Text))

Max Activating Token Index: 737

Text #8

Max Range: 0.8275. Min Range: -0.8275

Max Act: 0.6857. Min Act: -0.0000

Data Index: 207191 (C4 (Web Text))

Max Activating Token Index: 608

Text #9

Max Range: 0.8275. Min Range: -0.8275

Max Act: 0.6204. Min Act: -0.0000

Data Index: 165681 (C4 (Web Text))

Max Activating Token Index: 751

Text #10

Max Range: 0.8275. Min Range: -0.8275

Max Act: 0.6728. Min Act: -0.0000

Data Index: 570971 (C4 (Web Text))

Max Activating Token Index: 292

Text #11

Max Range: 0.8275. Min Range: -0.8275

Max Act: 0.6356. Min Act: -0.0000

Data Index: 989337 (C4 (Web Text))

Max Activating Token Index: 966

Text #12

Max Range: 0.8275. Min Range: -0.8275

Max Act: 0.6509. Min Act: -0.0000

Data Index: 647853 (C4 (Web Text))

Max Activating Token Index: 960

Text #13

Max Range: 0.8275. Min Range: -0.8275

Max Act: 0.6840. Min Act: -0.0000

Data Index: 651506 (C4 (Web Text))

Max Activating Token Index: 242

Text #14

Max Range: 0.8275. Min Range: -0.8275

Max Act: 0.6648. Min Act: -0.0000

Data Index: 978999 (C4 (Web Text))

Max Activating Token Index: 983

Text #15

Max Range: 0.8275. Min Range: -0.8275

Max Act: 0.6698. Min Act: -0.0000

Data Index: 953846 (C4 (Web Text))

Max Activating Token Index: 707

Text #16

Max Range: 0.8275. Min Range: -0.8275

Max Act: 0.6259. Min Act: -0.0000

Data Index: 1060152 (C4 (Web Text))

Max Activating Token Index: 918

Text #17

Max Range: 0.8275. Min Range: -0.8275

Max Act: 0.6032. Min Act: -0.0000

Data Index: 402332 (C4 (Web Text))

Max Activating Token Index: 841

Text #18

Max Range: 0.8275. Min Range: -0.8275

Max Act: 0.6264. Min Act: -0.0000

Data Index: 336835 (C4 (Web Text))

Max Activating Token Index: 635

Text #19

Max Range: 0.8275. Min Range: -0.8275

Max Act: 0.6238. Min Act: -0.0000

Data Index: 466861 (C4 (Web Text))

Max Activating Token Index: 690