Model: SoLU Model: 10 Layers, 5120 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 1102 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-10l')

Text #0

Max Range: 1.3730. Min Range: -1.3730

Max Act: 1.3730. Min Act: -0.0002

Data Index: 1033328 (C4 (Web Text))

Max Activating Token Index: 824

Text #1

Max Range: 1.3730. Min Range: -1.3730

Max Act: 1.2978. Min Act: -0.0003

Data Index: 1190510 (C4 (Web Text))

Max Activating Token Index: 411

Text #2

Max Range: 1.3730. Min Range: -1.3730

Max Act: 1.2809. Min Act: -0.0003

Data Index: 1214478 (C4 (Web Text))

Max Activating Token Index: 710

Text #3

Max Range: 1.3730. Min Range: -1.3730

Max Act: 1.3102. Min Act: -0.0002

Data Index: 460964 (C4 (Web Text))

Max Activating Token Index: 380

Text #4

Max Range: 1.3730. Min Range: -1.3730

Max Act: 1.2670. Min Act: -0.0002

Data Index: 248267 (C4 (Web Text))

Max Activating Token Index: 377

Text #5

Max Range: 1.3730. Min Range: -1.3730

Max Act: 1.2547. Min Act: -0.0003

Data Index: 335706 (C4 (Web Text))

Max Activating Token Index: 445

Text #6

Max Range: 1.3730. Min Range: -1.3730

Max Act: 1.1973. Min Act: -0.0003

Data Index: 725996 (C4 (Web Text))

Max Activating Token Index: 457

Text #7

Max Range: 1.3730. Min Range: -1.3730

Max Act: 1.2310. Min Act: -0.0002

Data Index: 865449 (C4 (Web Text))

Max Activating Token Index: 546

Text #8

Max Range: 1.3730. Min Range: -1.3730

Max Act: 1.2154. Min Act: -0.0003

Data Index: 568275 (C4 (Web Text))

Max Activating Token Index: 492

Text #9

Max Range: 1.3730. Min Range: -1.3730

Max Act: 1.1839. Min Act: -0.0003

Data Index: 617255 (C4 (Web Text))

Max Activating Token Index: 921

Text #10

Max Range: 1.3730. Min Range: -1.3730

Max Act: 1.1440. Min Act: -0.0003

Data Index: 373531 (C4 (Web Text))

Max Activating Token Index: 875

Text #11

Max Range: 1.3730. Min Range: -1.3730

Max Act: 1.0999. Min Act: -0.0003

Data Index: 1076348 (C4 (Web Text))

Max Activating Token Index: 175

Text #12

Max Range: 1.3730. Min Range: -1.3730

Max Act: 1.0541. Min Act: -0.0003

Data Index: 940757 (C4 (Web Text))

Max Activating Token Index: 41

Text #13

Max Range: 1.3730. Min Range: -1.3730

Max Act: 1.0738. Min Act: -0.0003

Data Index: 137271 (C4 (Web Text))

Max Activating Token Index: 728

Text #14

Max Range: 1.3730. Min Range: -1.3730

Max Act: 1.0341. Min Act: -0.0002

Data Index: 842071 (C4 (Web Text))

Max Activating Token Index: 189

Text #15

Max Range: 1.3730. Min Range: -1.3730

Max Act: 0.9907. Min Act: -0.0002

Data Index: 793375 (C4 (Web Text))

Max Activating Token Index: 281

Text #16

Max Range: 1.3730. Min Range: -1.3730

Max Act: 0.9891. Min Act: -0.0002

Data Index: 819934 (C4 (Web Text))

Max Activating Token Index: 162

Text #17

Max Range: 1.3730. Min Range: -1.3730

Max Act: 1.0045. Min Act: -0.0002

Data Index: 1120388 (C4 (Web Text))

Max Activating Token Index: 568

Text #18

Max Range: 1.3730. Min Range: -1.3730

Max Act: 0.9823. Min Act: -0.0002

Data Index: 100217 (C4 (Web Text))

Max Activating Token Index: 609

Text #19

Max Range: 1.3730. Min Range: -1.3730

Max Act: 0.9708. Min Act: -0.0002

Data Index: 369921 (C4 (Web Text))

Max Activating Token Index: 96