Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 363 in Layer 5

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 0.1787. Min Range: -0.1787

Max Act: 0.1787. Min Act: -0.0000

Data Index: 132401 (C4 (Web Text))

Max Activating Token Index: 357

Text #1

Max Range: 0.1787. Min Range: -0.1787

Max Act: 0.1428. Min Act: -0.0000

Data Index: 429735 (C4 (Web Text))

Max Activating Token Index: 659

Text #2

Max Range: 0.1787. Min Range: -0.1787

Max Act: 0.1562. Min Act: -0.0000

Data Index: 736476 (C4 (Web Text))

Max Activating Token Index: 91

Text #3

Max Range: 0.1787. Min Range: -0.1787

Max Act: 0.1451. Min Act: -0.0000

Data Index: 833750 (C4 (Web Text))

Max Activating Token Index: 991

Text #4

Max Range: 0.1787. Min Range: -0.1787

Max Act: 0.1300. Min Act: -0.0000

Data Index: 675738 (C4 (Web Text))

Max Activating Token Index: 139

Text #5

Max Range: 0.1787. Min Range: -0.1787

Max Act: 0.1367. Min Act: -0.0000

Data Index: 1181297 (C4 (Web Text))

Max Activating Token Index: 106

Text #6

Max Range: 0.1787. Min Range: -0.1787

Max Act: 0.1325. Min Act: -0.0000

Data Index: 730996 (C4 (Web Text))

Max Activating Token Index: 1022

Text #7

Max Range: 0.1787. Min Range: -0.1787

Max Act: 0.1291. Min Act: -0.0000

Data Index: 990713 (C4 (Web Text))

Max Activating Token Index: 408

Text #8

Max Range: 0.1787. Min Range: -0.1787

Max Act: 0.1273. Min Act: -0.0000

Data Index: 105342 (C4 (Web Text))

Max Activating Token Index: 715

Text #9

Max Range: 0.1787. Min Range: -0.1787

Max Act: 0.1312. Min Act: -0.0000

Data Index: 402810 (C4 (Web Text))

Max Activating Token Index: 301

Text #10

Max Range: 0.1787. Min Range: -0.1787

Max Act: 0.1235. Min Act: -0.0000

Data Index: 1273291 (C4 (Web Text))

Max Activating Token Index: 263

Text #11

Max Range: 0.1787. Min Range: -0.1787

Max Act: 0.1228. Min Act: -0.0000

Data Index: 680909 (C4 (Web Text))

Max Activating Token Index: 713

Text #12

Max Range: 0.1787. Min Range: -0.1787

Max Act: 0.1233. Min Act: -0.0001

Data Index: 1330291 (C4 (Web Text))

Max Activating Token Index: 622

Text #13

Max Range: 0.1787. Min Range: -0.1787

Max Act: 0.1220. Min Act: -0.0000

Data Index: 85176 (C4 (Web Text))

Max Activating Token Index: 999

Text #14

Max Range: 0.1787. Min Range: -0.1787

Max Act: 0.1249. Min Act: -0.0000

Data Index: 1218135 (C4 (Web Text))

Max Activating Token Index: 1013

Text #15

Max Range: 0.1787. Min Range: -0.1787

Max Act: 0.1112. Min Act: -0.0000

Data Index: 176501 (C4 (Web Text))

Max Activating Token Index: 194

Text #16

Max Range: 0.1787. Min Range: -0.1787

Max Act: 0.1119. Min Act: -0.0000

Data Index: 824971 (C4 (Web Text))

Max Activating Token Index: 760

Text #17

Max Range: 0.1787. Min Range: -0.1787

Max Act: 0.1194. Min Act: -0.0000

Data Index: 960673 (C4 (Web Text))

Max Activating Token Index: 393

Text #18

Max Range: 0.1787. Min Range: -0.1787

Max Act: 0.1182. Min Act: -0.0000

Data Index: 95846 (C4 (Web Text))

Max Activating Token Index: 15

Text #19

Max Range: 0.1787. Min Range: -0.1787

Max Act: 0.1177. Min Act: -0.0000

Data Index: 654309 (C4 (Web Text))

Max Activating Token Index: 36