Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: The Pile

Neuron 4303 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l-pile')

Text #0

Max Range: 5.2304. Min Range: -5.2304

Max Act: 5.2304. Min Act: -0.0000

Data Index: 68918 (The Pile)

Max Activating Token Index: 768

Text #1

Max Range: 5.2304. Min Range: -5.2304

Max Act: 4.3371. Min Act: -0.0000

Data Index: 251352 (The Pile)

Max Activating Token Index: 819

Text #2

Max Range: 5.2304. Min Range: -5.2304

Max Act: 4.1343. Min Act: -0.0000

Data Index: 1159115 (The Pile)

Max Activating Token Index: 154

Text #3

Max Range: 5.2304. Min Range: -5.2304

Max Act: 4.3019. Min Act: -0.0000

Data Index: 169861 (The Pile)

Max Activating Token Index: 540

Text #4

Max Range: 5.2304. Min Range: -5.2304

Max Act: 3.8778. Min Act: -0.0000

Data Index: 1010856 (The Pile)

Max Activating Token Index: 486

Text #5

Max Range: 5.2304. Min Range: -5.2304

Max Act: 4.1204. Min Act: -0.0000

Data Index: 188625 (The Pile)

Max Activating Token Index: 687

Text #6

Max Range: 5.2304. Min Range: -5.2304

Max Act: 3.9286. Min Act: -0.0000

Data Index: 1518329 (The Pile)

Max Activating Token Index: 626

Text #7

Max Range: 5.2304. Min Range: -5.2304

Max Act: 3.1027. Min Act: -0.0000

Data Index: 1059523 (The Pile)

Max Activating Token Index: 854

Text #8

Max Range: 5.2304. Min Range: -5.2304

Max Act: 3.9892. Min Act: -0.0000

Data Index: 1680897 (The Pile)

Max Activating Token Index: 571

Text #9

Max Range: 5.2304. Min Range: -5.2304

Max Act: 3.7220. Min Act: -0.0000

Data Index: 1095668 (The Pile)

Max Activating Token Index: 370

Text #10

Max Range: 5.2304. Min Range: -5.2304

Max Act: 3.1988. Min Act: -0.0000

Data Index: 926092 (The Pile)

Max Activating Token Index: 97

Text #11

Max Range: 5.2304. Min Range: -5.2304

Max Act: 3.6876. Min Act: -0.0000

Data Index: 194510 (The Pile)

Max Activating Token Index: 131

Text #12

Max Range: 5.2304. Min Range: -5.2304

Max Act: 3.5122. Min Act: -0.0000

Data Index: 1208874 (The Pile)

Max Activating Token Index: 1021

Text #13

Max Range: 5.2304. Min Range: -5.2304

Max Act: 3.8026. Min Act: -0.0000

Data Index: 901922 (The Pile)

Max Activating Token Index: 351

Text #14

Max Range: 5.2304. Min Range: -5.2304

Max Act: 3.3019. Min Act: -0.0000

Data Index: 1736335 (The Pile)

Max Activating Token Index: 548

Text #15

Max Range: 5.2304. Min Range: -5.2304

Max Act: 3.4231. Min Act: -0.0000

Data Index: 1062261 (The Pile)

Max Activating Token Index: 208

Text #16

Max Range: 5.2304. Min Range: -5.2304

Max Act: 3.3394. Min Act: -0.0000

Data Index: 578821 (The Pile)

Max Activating Token Index: 827

Text #17

Max Range: 5.2304. Min Range: -5.2304

Max Act: 3.4692. Min Act: -0.0000

Data Index: 345741 (The Pile)

Max Activating Token Index: 731

Text #18

Max Range: 5.2304. Min Range: -5.2304

Max Act: 3.1664. Min Act: -0.0000

Data Index: 1035460 (The Pile)

Max Activating Token Index: 337

Text #19

Max Range: 5.2304. Min Range: -5.2304

Max Act: 3.3200. Min Act: -0.0000

Data Index: 1479505 (The Pile)

Max Activating Token Index: 286