Model: SoLU Model: 12 Layers, 6144 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 3569 in Layer 1

Load this data into an Interactive Neuroscope

See Documentation here

Hooked Transformer Loading: HookedTransformer.from_pretrained('solu-12l')

Click the toggle to see the full text!

Text #0

Max Range: 0.3794. Min Range: -0.3794

Max Act: 0.3794. Min Act: -0.0002

Data Index: 930785 (C4 (Web Text))

Max Activating Token Index: 311

Text #1

Max Range: 0.3794. Min Range: -0.3794

Max Act: 0.3727. Min Act: -0.0002

Data Index: 787057 (C4 (Web Text))

Max Activating Token Index: 552

Text #2

Max Range: 0.3794. Min Range: -0.3794

Max Act: 0.3486. Min Act: -0.0002

Data Index: 175250 (C4 (Web Text))

Max Activating Token Index: 569

Text #3

Max Range: 0.3794. Min Range: -0.3794

Max Act: 0.3344. Min Act: -0.0002

Data Index: 1312233 (C4 (Web Text))

Max Activating Token Index: 367

Text #4

Max Range: 0.3794. Min Range: -0.3794

Max Act: 0.3322. Min Act: -0.0002

Data Index: 266869 (C4 (Web Text))

Max Activating Token Index: 57

Text #5

Max Range: 0.3794. Min Range: -0.3794

Max Act: 0.3170. Min Act: -0.0002

Data Index: 1336766 (C4 (Web Text))

Max Activating Token Index: 201

Text #6

Max Range: 0.3794. Min Range: -0.3794

Max Act: 0.3101. Min Act: -0.0002

Data Index: 666087 (C4 (Web Text))

Max Activating Token Index: 724

Text #7

Max Range: 0.3794. Min Range: -0.3794

Max Act: 0.3009. Min Act: -0.0002

Data Index: 366772 (C4 (Web Text))

Max Activating Token Index: 509

Text #8

Max Range: 0.3794. Min Range: -0.3794

Max Act: 0.2995. Min Act: -0.0002

Data Index: 613169 (C4 (Web Text))

Max Activating Token Index: 329

Text #9

Max Range: 0.3794. Min Range: -0.3794

Max Act: 0.2974. Min Act: -0.0003

Data Index: 301252 (C4 (Web Text))

Max Activating Token Index: 754

Text #10

Max Range: 0.3794. Min Range: -0.3794

Max Act: 0.2950. Min Act: -0.0002

Data Index: 846694 (C4 (Web Text))

Max Activating Token Index: 143

Text #11

Max Range: 0.3794. Min Range: -0.3794

Max Act: 0.2921. Min Act: -0.0002

Data Index: 585268 (C4 (Web Text))

Max Activating Token Index: 767

Text #12

Max Range: 0.3794. Min Range: -0.3794

Max Act: 0.2923. Min Act: -0.0002

Data Index: 258431 (C4 (Web Text))

Max Activating Token Index: 228

Text #13

Max Range: 0.3794. Min Range: -0.3794

Max Act: 0.2852. Min Act: -0.0002

Data Index: 1217371 (C4 (Web Text))

Max Activating Token Index: 442

Text #14

Max Range: 0.3794. Min Range: -0.3794

Max Act: 0.2823. Min Act: -0.0002

Data Index: 1119826 (C4 (Web Text))

Max Activating Token Index: 455

Text #15

Max Range: 0.3794. Min Range: -0.3794

Max Act: 0.2844. Min Act: -0.0002

Data Index: 445 (C4 (Web Text))

Max Activating Token Index: 319

Text #16

Max Range: 0.3794. Min Range: -0.3794

Max Act: 0.2823. Min Act: -0.0002

Data Index: 1007794 (C4 (Web Text))

Max Activating Token Index: 655

Text #17

Max Range: 0.3794. Min Range: -0.3794

Max Act: 0.2862. Min Act: -0.0002

Data Index: 706019 (C4 (Web Text))

Max Activating Token Index: 355

Text #18

Max Range: 0.3794. Min Range: -0.3794

Max Act: 0.2836. Min Act: -0.0002

Data Index: 850715 (C4 (Web Text))

Max Activating Token Index: 519

Text #19

Max Range: 0.3794. Min Range: -0.3794

Max Act: 0.2813. Min Act: -0.0002

Data Index: 1349969 (C4 (Web Text))

Max Activating Token Index: 944