Model: SoLU Model: 6 Layers, 3072 Neurons per Layer

Dataset: 80% C4 (Web Text) and 20% Python Code

Neuron 2782 in Layer 3

Load this data into an Interactive Neuroscope

See Documentation here

Transformer Lens Loading: HookedTransformer.from_pretrained('solu-6l')

Text #0

Max Range: 0.5962. Min Range: -0.5962

Max Act: 0.5962. Min Act: -0.0001

Data Index: 1379567 (Python Code)

Max Activating Token Index: 804

Text #1

Max Range: 0.5962. Min Range: -0.5962

Max Act: 0.5678. Min Act: -0.0001

Data Index: 1438433 (Python Code)

Max Activating Token Index: 850

Text #2

Max Range: 0.5962. Min Range: -0.5962

Max Act: 0.5439. Min Act: -0.0001

Data Index: 1443339 (Python Code)

Max Activating Token Index: 199

Text #3

Max Range: 0.5962. Min Range: -0.5962

Max Act: 0.5735. Min Act: -0.0001

Data Index: 1654754 (Python Code)

Max Activating Token Index: 764

Text #4

Max Range: 0.5962. Min Range: -0.5962

Max Act: 0.5135. Min Act: -0.0001

Data Index: 1631482 (Python Code)

Max Activating Token Index: 146

Text #5

Max Range: 0.5962. Min Range: -0.5962

Max Act: 0.5623. Min Act: -0.0001

Data Index: 1432276 (Python Code)

Max Activating Token Index: 829

Text #6

Max Range: 0.5962. Min Range: -0.5962

Max Act: 0.5360. Min Act: -0.0001

Data Index: 1587293 (Python Code)

Max Activating Token Index: 885

Text #7

Max Range: 0.5962. Min Range: -0.5962

Max Act: 0.4541. Min Act: -0.0001

Data Index: 1640424 (Python Code)

Max Activating Token Index: 310

Text #8

Max Range: 0.5962. Min Range: -0.5962

Max Act: 0.4703. Min Act: -0.0001

Data Index: 1478680 (Python Code)

Max Activating Token Index: 754

Text #9

Max Range: 0.5962. Min Range: -0.5962

Max Act: 0.4971. Min Act: -0.0001

Data Index: 1490004 (Python Code)

Max Activating Token Index: 957

Text #10

Max Range: 0.5962. Min Range: -0.5962

Max Act: 0.5210. Min Act: -0.0001

Data Index: 1530077 (Python Code)

Max Activating Token Index: 229

Text #11

Max Range: 0.5962. Min Range: -0.5962

Max Act: 0.4860. Min Act: -0.0001

Data Index: 1462718 (Python Code)

Max Activating Token Index: 84

Text #12

Max Range: 0.5962. Min Range: -0.5962

Max Act: 0.4960. Min Act: -0.0001

Data Index: 1375961 (Python Code)

Max Activating Token Index: 100

Text #13

Max Range: 0.5962. Min Range: -0.5962

Max Act: 0.4861. Min Act: -0.0001

Data Index: 1540493 (Python Code)

Max Activating Token Index: 761

Text #14

Max Range: 0.5962. Min Range: -0.5962

Max Act: 0.4959. Min Act: -0.0001

Data Index: 1373095 (Python Code)

Max Activating Token Index: 51

Text #15

Max Range: 0.5962. Min Range: -0.5962

Max Act: 0.4751. Min Act: -0.0001

Data Index: 1509377 (Python Code)

Max Activating Token Index: 740

Text #16

Max Range: 0.5962. Min Range: -0.5962

Max Act: 0.4871. Min Act: -0.0001

Data Index: 1374491 (Python Code)

Max Activating Token Index: 546

Text #17

Max Range: 0.5962. Min Range: -0.5962

Max Act: 0.4750. Min Act: -0.0001

Data Index: 1464042 (Python Code)

Max Activating Token Index: 111

Text #18

Max Range: 0.5962. Min Range: -0.5962

Max Act: 0.5019. Min Act: -0.0001

Data Index: 1528852 (Python Code)

Max Activating Token Index: 376

Text #19

Max Range: 0.5962. Min Range: -0.5962

Max Act: 0.5047. Min Act: -0.0001

Data Index: 1633445 (Python Code)

Max Activating Token Index: 188