Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2047 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 6.4369. Min Range: -6.4369

Max Act: 6.4369. Min Act: -0.0000

Data Index: 63757 (C4 (Web Text))

Max Activating Token Index: 180

Text #1

Max Range: 6.4369. Min Range: -6.4369

Max Act: 6.4263. Min Act: -0.0000

Data Index: 921799 (C4 (Web Text))

Max Activating Token Index: 95

Text #2

Max Range: 6.4369. Min Range: -6.4369

Max Act: 5.6332. Min Act: -0.0000

Data Index: 347521 (C4 (Web Text))

Max Activating Token Index: 296

Text #3

Max Range: 6.4369. Min Range: -6.4369

Max Act: 5.9532. Min Act: -0.0000

Data Index: 214679 (C4 (Web Text))

Max Activating Token Index: 290

Text #4

Max Range: 6.4369. Min Range: -6.4369

Max Act: 5.7375. Min Act: -0.0000

Data Index: 215883 (C4 (Web Text))

Max Activating Token Index: 99

Text #5

Max Range: 6.4369. Min Range: -6.4369

Max Act: 5.5175. Min Act: -0.0000

Data Index: 159097 (C4 (Web Text))

Max Activating Token Index: 180

Text #6

Max Range: 6.4369. Min Range: -6.4369

Max Act: 5.6760. Min Act: -0.0000

Data Index: 391784 (C4 (Web Text))

Max Activating Token Index: 356

Text #7

Max Range: 6.4369. Min Range: -6.4369

Max Act: 5.8341. Min Act: -0.0000

Data Index: 277389 (C4 (Web Text))

Max Activating Token Index: 307

Text #8

Max Range: 6.4369. Min Range: -6.4369

Max Act: 5.4160. Min Act: -0.0000

Data Index: 637445 (C4 (Web Text))

Max Activating Token Index: 585

Text #9

Max Range: 6.4369. Min Range: -6.4369

Max Act: 5.5333. Min Act: -0.0000

Data Index: 217782 (C4 (Web Text))

Max Activating Token Index: 790

Text #10

Max Range: 6.4369. Min Range: -6.4369

Max Act: 5.3482. Min Act: -0.0000

Data Index: 958965 (C4 (Web Text))

Max Activating Token Index: 49

Text #11

Max Range: 6.4369. Min Range: -6.4369

Max Act: 5.4924. Min Act: -0.0000

Data Index: 234398 (C4 (Web Text))

Max Activating Token Index: 59

Text #12

Max Range: 6.4369. Min Range: -6.4369

Max Act: 5.3873. Min Act: -0.0000

Data Index: 20106 (C4 (Web Text))

Max Activating Token Index: 562

Text #13

Max Range: 6.4369. Min Range: -6.4369

Max Act: 5.3319. Min Act: -0.0000

Data Index: 97229 (C4 (Web Text))

Max Activating Token Index: 914

Text #14

Max Range: 6.4369. Min Range: -6.4369

Max Act: 5.6794. Min Act: -0.0000

Data Index: 954851 (C4 (Web Text))

Max Activating Token Index: 687

Text #15

Max Range: 6.4369. Min Range: -6.4369

Max Act: 5.3211. Min Act: -0.0000

Data Index: 301398 (C4 (Web Text))

Max Activating Token Index: 832

Text #16

Max Range: 6.4369. Min Range: -6.4369

Max Act: 5.3019. Min Act: -0.0000

Data Index: 724668 (C4 (Web Text))

Max Activating Token Index: 21

Text #17

Max Range: 6.4369. Min Range: -6.4369

Max Act: 5.2470. Min Act: -0.0000

Data Index: 769478 (C4 (Web Text))

Max Activating Token Index: 392

Text #18

Max Range: 6.4369. Min Range: -6.4369

Max Act: 5.2144. Min Act: -0.0000

Data Index: 279278 (C4 (Web Text))

Max Activating Token Index: 464

Text #19

Max Range: 6.4369. Min Range: -6.4369

Max Act: 5.1282. Min Act: -0.0000

Data Index: 1048537 (C4 (Web Text))

Max Activating Token Index: 877