Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1583 in Layer 9

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.5315. Min Range: -1.5315

Max Act: 1.5315. Min Act: 0.0000

Data Index: 940841 (C4 (Web Text))

Max Activating Token Index: 436

Text #1

Max Range: 1.5315. Min Range: -1.5315

Max Act: 1.1179. Min Act: -0.0000

Data Index: 340494 (C4 (Web Text))

Max Activating Token Index: 1002

Text #2

Max Range: 1.5315. Min Range: -1.5315

Max Act: 0.9769. Min Act: -0.0000

Data Index: 88360 (C4 (Web Text))

Max Activating Token Index: 345

Text #3

Max Range: 1.5315. Min Range: -1.5315

Max Act: 0.9258. Min Act: -0.0000

Data Index: 986382 (C4 (Web Text))

Max Activating Token Index: 538

Text #4

Max Range: 1.5315. Min Range: -1.5315

Max Act: 0.9437. Min Act: 0.0000

Data Index: 743564 (C4 (Web Text))

Max Activating Token Index: 458

Text #5

Max Range: 1.5315. Min Range: -1.5315

Max Act: 0.9681. Min Act: 0.0000

Data Index: 535923 (C4 (Web Text))

Max Activating Token Index: 527

Text #6

Max Range: 1.5315. Min Range: -1.5315

Max Act: 0.9188. Min Act: -0.0000

Data Index: 623474 (C4 (Web Text))

Max Activating Token Index: 158

Text #7

Max Range: 1.5315. Min Range: -1.5315

Max Act: 0.7847. Min Act: -0.0000

Data Index: 582680 (C4 (Web Text))

Max Activating Token Index: 725

Text #8

Max Range: 1.5315. Min Range: -1.5315

Max Act: 1.0053. Min Act: -0.0000

Data Index: 789607 (C4 (Web Text))

Max Activating Token Index: 645

Text #9

Max Range: 1.5315. Min Range: -1.5315

Max Act: 0.8510. Min Act: -0.0000

Data Index: 290616 (C4 (Web Text))

Max Activating Token Index: 462

Text #10

Max Range: 1.5315. Min Range: -1.5315

Max Act: 0.9011. Min Act: -0.0000

Data Index: 93399 (C4 (Web Text))

Max Activating Token Index: 381

Text #11

Max Range: 1.5315. Min Range: -1.5315

Max Act: 0.7541. Min Act: -0.0000

Data Index: 880682 (C4 (Web Text))

Max Activating Token Index: 302

Text #12

Max Range: 1.5315. Min Range: -1.5315

Max Act: 0.8678. Min Act: -0.0000

Data Index: 271891 (C4 (Web Text))

Max Activating Token Index: 510

Text #13

Max Range: 1.5315. Min Range: -1.5315

Max Act: 0.6566. Min Act: -0.0000

Data Index: 1073686 (C4 (Web Text))

Max Activating Token Index: 817

Text #14

Max Range: 1.5315. Min Range: -1.5315

Max Act: 0.7365. Min Act: -0.0000

Data Index: 117063 (C4 (Web Text))

Max Activating Token Index: 301

Text #15

Max Range: 1.5315. Min Range: -1.5315

Max Act: 0.7943. Min Act: 0.0000

Data Index: 978654 (C4 (Web Text))

Max Activating Token Index: 953

Text #16

Max Range: 1.5315. Min Range: -1.5315

Max Act: 0.7282. Min Act: -0.0000

Data Index: 553564 (C4 (Web Text))

Max Activating Token Index: 653

Text #17

Max Range: 1.5315. Min Range: -1.5315

Max Act: 0.6738. Min Act: -0.0000

Data Index: 1110779 (C4 (Web Text))

Max Activating Token Index: 564

Text #18

Max Range: 1.5315. Min Range: -1.5315

Max Act: 0.7290. Min Act: 0.0000

Data Index: 571329 (C4 (Web Text))

Max Activating Token Index: 639

Text #19

Max Range: 1.5315. Min Range: -1.5315

Max Act: 0.6610. Min Act: -0.0000

Data Index: 39426 (C4 (Web Text))

Max Activating Token Index: 754