Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 49 in Layer 6

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-12l')

Text #0

Max Range: 2.7970. Min Range: -2.7970

Max Act: 2.7099. Min Act: -0.0000

Data Index: 282279 (C4 (Web Text))

Max Activating Token Index: 32

Text #1

Max Range: 2.7970. Min Range: -2.7970

Max Act: 2.7970. Min Act: -0.0000

Data Index: 615860 (C4 (Web Text))

Max Activating Token Index: 679

Text #2

Max Range: 2.7970. Min Range: -2.7970

Max Act: 2.4598. Min Act: -0.0000

Data Index: 620683 (C4 (Web Text))

Max Activating Token Index: 839

Text #3

Max Range: 2.7970. Min Range: -2.7970

Max Act: 2.3013. Min Act: -0.0000

Data Index: 828074 (C4 (Web Text))

Max Activating Token Index: 319

Text #4

Max Range: 2.7970. Min Range: -2.7970

Max Act: 2.5486. Min Act: -0.0000

Data Index: 1347951 (C4 (Web Text))

Max Activating Token Index: 698

Text #5

Max Range: 2.7970. Min Range: -2.7970

Max Act: 2.3847. Min Act: -0.0000

Data Index: 607718 (C4 (Web Text))

Max Activating Token Index: 310

Text #6

Max Range: 2.7970. Min Range: -2.7970

Max Act: 2.6177. Min Act: -0.0000

Data Index: 624165 (C4 (Web Text))

Max Activating Token Index: 812

Text #7

Max Range: 2.7970. Min Range: -2.7970

Max Act: 2.2489. Min Act: -0.0000

Data Index: 914329 (C4 (Web Text))

Max Activating Token Index: 300

Text #8

Max Range: 2.7970. Min Range: -2.7970

Max Act: 2.4811. Min Act: -0.0000

Data Index: 655499 (C4 (Web Text))

Max Activating Token Index: 492

Text #9

Max Range: 2.7970. Min Range: -2.7970

Max Act: 2.3291. Min Act: -0.0000

Data Index: 250528 (C4 (Web Text))

Max Activating Token Index: 636

Text #10

Max Range: 2.7970. Min Range: -2.7970

Max Act: 2.4224. Min Act: -0.0000

Data Index: 239311 (C4 (Web Text))

Max Activating Token Index: 522

Text #11

Max Range: 2.7970. Min Range: -2.7970

Max Act: 2.0467. Min Act: -0.0000

Data Index: 853908 (C4 (Web Text))

Max Activating Token Index: 281

Text #12

Max Range: 2.7970. Min Range: -2.7970

Max Act: 2.2157. Min Act: -0.0000

Data Index: 16302 (C4 (Web Text))

Max Activating Token Index: 605

Text #13

Max Range: 2.7970. Min Range: -2.7970

Max Act: 2.2179. Min Act: -0.0000

Data Index: 66424 (C4 (Web Text))

Max Activating Token Index: 816

Text #14

Max Range: 2.7970. Min Range: -2.7970

Max Act: 2.1350. Min Act: -0.0000

Data Index: 1215938 (C4 (Web Text))

Max Activating Token Index: 621

Text #15

Max Range: 2.7970. Min Range: -2.7970

Max Act: 1.9302. Min Act: -0.0000

Data Index: 635681 (C4 (Web Text))

Max Activating Token Index: 808

Text #16

Max Range: 2.7970. Min Range: -2.7970

Max Act: 2.2437. Min Act: -0.0000

Data Index: 508477 (C4 (Web Text))

Max Activating Token Index: 462

Text #17

Max Range: 2.7970. Min Range: -2.7970

Max Act: 2.2314. Min Act: -0.0000

Data Index: 1224606 (C4 (Web Text))

Max Activating Token Index: 156

Text #18

Max Range: 2.7970. Min Range: -2.7970

Max Act: 2.2134. Min Act: -0.0000

Data Index: 856959 (C4 (Web Text))

Max Activating Token Index: 270

Text #19

Max Range: 2.7970. Min Range: -2.7970

Max Act: 1.8829. Min Act: -0.0000

Data Index: 927291 (C4 (Web Text))

Max Activating Token Index: 798