Model: SoLU Model: 8 Layers, 4096 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3272 in Layer 7

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-8l')

Text #0

Max Range: 0.6631. Min Range: -0.6631

Max Act: 0.6631. Min Act: -0.0001

Data Index: 294477 (C4 (Web Text))

Max Activating Token Index: 959

Text #1

Max Range: 0.6631. Min Range: -0.6631

Max Act: 0.6166. Min Act: 0.0000

Data Index: 759618 (C4 (Web Text))

Max Activating Token Index: 165

Text #2

Max Range: 0.6631. Min Range: -0.6631

Max Act: 0.5629. Min Act: -0.0000

Data Index: 914096 (C4 (Web Text))

Max Activating Token Index: 72

Text #3

Max Range: 0.6631. Min Range: -0.6631

Max Act: 0.5456. Min Act: -0.0001

Data Index: 507891 (C4 (Web Text))

Max Activating Token Index: 568

Text #4

Max Range: 0.6631. Min Range: -0.6631

Max Act: 0.5460. Min Act: -0.0001

Data Index: 1164574 (C4 (Web Text))

Max Activating Token Index: 37

Text #5

Max Range: 0.6631. Min Range: -0.6631

Max Act: 0.5256. Min Act: -0.0001

Data Index: 1425048 (Python Code)

Max Activating Token Index: 713

Text #6

Max Range: 0.6631. Min Range: -0.6631

Max Act: 0.4785. Min Act: 0.0001

Data Index: 1581167 (Python Code)

Max Activating Token Index: 234

Text #7

Max Range: 0.6631. Min Range: -0.6631

Max Act: 0.4911. Min Act: -0.0000

Data Index: 883013 (C4 (Web Text))

Max Activating Token Index: 525

Text #8

Max Range: 0.6631. Min Range: -0.6631

Max Act: 0.4906. Min Act: -0.0000

Data Index: 1041257 (C4 (Web Text))

Max Activating Token Index: 229

Text #9

Max Range: 0.6631. Min Range: -0.6631

Max Act: 0.4708. Min Act: 0.0001

Data Index: 1065322 (C4 (Web Text))

Max Activating Token Index: 516

Text #10

Max Range: 0.6631. Min Range: -0.6631

Max Act: 0.4925. Min Act: -0.0000

Data Index: 327728 (C4 (Web Text))

Max Activating Token Index: 451

Text #11

Max Range: 0.6631. Min Range: -0.6631

Max Act: 0.4565. Min Act: -0.0001

Data Index: 816365 (C4 (Web Text))

Max Activating Token Index: 454

Text #12

Max Range: 0.6631. Min Range: -0.6631

Max Act: 0.4328. Min Act: 0.0001

Data Index: 810392 (C4 (Web Text))

Max Activating Token Index: 187

Text #13

Max Range: 0.6631. Min Range: -0.6631

Max Act: 0.4336. Min Act: -0.0000

Data Index: 488595 (C4 (Web Text))

Max Activating Token Index: 940

Text #14

Max Range: 0.6631. Min Range: -0.6631

Max Act: 0.4214. Min Act: 0.0001

Data Index: 1110273 (C4 (Web Text))

Max Activating Token Index: 448

Text #15

Max Range: 0.6631. Min Range: -0.6631

Max Act: 0.4274. Min Act: -0.0000

Data Index: 1264872 (C4 (Web Text))

Max Activating Token Index: 933

Text #16

Max Range: 0.6631. Min Range: -0.6631

Max Act: 0.4265. Min Act: -0.0001

Data Index: 531792 (C4 (Web Text))

Max Activating Token Index: 764

Text #17

Max Range: 0.6631. Min Range: -0.6631

Max Act: 0.4306. Min Act: 0.0001

Data Index: 1315395 (C4 (Web Text))

Max Activating Token Index: 183

Text #18

Max Range: 0.6631. Min Range: -0.6631

Max Act: 0.4023. Min Act: -0.0000

Data Index: 1187676 (C4 (Web Text))

Max Activating Token Index: 509

Text #19

Max Range: 0.6631. Min Range: -0.6631

Max Act: 0.3996. Min Act: -0.0001

Data Index: 608886 (C4 (Web Text))

Max Activating Token Index: 189